Skip to documentation
eli.aidocs

Ports

Approvals & feedback

Two human-in-the-loop surfaces: approvals pause a run before a risky tool executes, and feedback captures your judgment on answers so agents can improve. Both are durable, workspace-scoped records.

Approving gated tool calls

When an agent calls a tool flagged requiresApproval (all MCP tools in v1), the run suspends: its status becomes suspended and a run_approvals row is persisted with the tool name and exact input. Pending approvals collect at /w/<workspace>/approvals.

  1. Review the request

    The approval card shows the agent, the tool name, and the full input payload the model wants to send. You are approving this specific call with this specific input.

  2. Decide

    Three verbs: allow, deny (optionally with a note), or edit & approve — modify the tool input JSON and the resumed call executes yourpayload while the agent's original stays on the approval row for audit. An allowed call executes and the run continues; a denied call returns an error result the model can react to. Every decision writes an audit row in the same transaction.

  3. The run resumes

    Resumption reads the persisted cursor and picks up exactly where it paused — even hours later or across a process restart.

Durable by construction

Because the full run state is persisted at suspend, the human can take as long as they need. Approval decisions are also captured as feedback signal. If the workspace sets an approval SLA (hitl.approvalSlaHours), a pending approval past due raises one deduped approval.overdue alert — escalation, never auto-decision or expiry.

Autopilot and retrospective review

Not every gated call needs a human up front. The workspace's human-review policy (Settings → Operational) chooses, per tool and optionally per agent, between require (suspend — the default) and autopilot (execute immediately). Three guardrails keep autopilot honest: every autopiloted call is logged and risk-scored (deterministic, named factors — external tool, write tool, payload size, outbound addresses, and the tool's own overturn history); calls scoring above the workspace's risk ceiling escalate to a normal suspension anyway (audited with the score and factors); and a configurable sample lands in the Autopilot reviewstab on this page for a retrospective verdict. Marking a call "overturn" is audited and raises that tool's historical risk factor, so autopilot tightens exactly where humans disagree with it. Knowledge-base writes (kb_propose_note) are a platform floor and always require review.

approve over the API (editedInput is optional — the edit verb)bash
curl -L -X POST https://eli.ai/api/v1/runs/$RUN_ID/approve \
  -H "Authorization: Bearer $ELI_KEY" \
  -H "Content-Type: application/json" \
  -d '{"approvalId":"01J…","decision":"allow","note":"narrowed the recipient list",
       "editedInput":{"to":"one-client@example.com","body":"reviewed copy"}}'

The in-app equivalent is POST /api/w/:workspaceId/approvals/:id; pending approvals list at GET /api/w/:workspaceId/approvals.

Capturing feedback

Feedback attaches to a run or a message and is the raw signal that drives the improvement loop. Five kinds:

  • thumb — { thumb: "up" | "down" }.
  • rating — { rating: 1..5 }.
  • correction — a free-text correction of what the answer got wrong.
  • edit — an edited final output.
  • citation_flag— a "wrong source" flag on a specific citation.
attach feedback to a runbash
curl -L -X POST https://eli.ai/api/v1/runs/$RUN_ID/feedback \
  -H "Authorization: Bearer $ELI_KEY" \
  -H "Content-Type: application/json" \
  -d '{"kind":"rating","value":{"rating":4},"note":"missed the Q3 caveat"}'

From feedback to improvement

Feedback feeds three channels, all reusing the evals substrate:

  • Golden-set promotion. A corrected answer becomes a golden item (question + atomic assertions + required citations) after human approval, feeding automated eval runs.
  • Few-shot curation. High-rated exchanges become use-case-tagged few-shot examples, selected at prompt-assembly time.
  • Agent revision. Aggregated failure patterns inform a new immutable agent version, measured against the golden set before rollout.

The full loop is described in Evals & drift. The in-app feedback endpoint is POST /api/w/:workspaceId/feedback.