Skip to documentation
eli.aidocs

Crucible

Golden sets & eval runs

Evals turn "it feels better" into a number. Author a golden set of questions with expected atomic assertions, run them through the real pipeline with a cost estimate up front, read per-tag metrics, and A/B-diff two configs. The evals workspace is at /w/<workspace>/evals. The model is in Evals & drift.

Author golden items

  1. Write a question and assertions

    A golden item is a question plus must / shouldassertions — atomic, independently verifiable claims. "The FY26 budget freezes Operations headcount" is one assertion; wording is free.

  2. Anchor citations

    Attach the documents (and quoted snippets) an answer should cite. Anchors resolve to chunks at run time, so they survive re-ingestion.

  3. Tag it

    Add bu:* and topic:* tags so metrics break down by business unit and topic — the same tags the coverage matrix overlays.

Promote from an answer

Instead of writing assertions by hand, click "promote" on a good chat/agent answer. The judge model decomposes it into candidate atomic assertions; you edit and approve before it's saved (never auto-saved). Endpoint: POST /api/w/:workspaceId/golden/promote.

Run the set

  1. Pick a config

    An eval config pins the target (which model/prompt) and the judge slot separately. You can run the whole set or a subset of items, at n=1 (smoke) or n=3 (regression-grade).

  2. Confirm the cost estimate

    A pre-run gate shows the estimated cost and asks for explicit confirmation — no surprise spend.

  3. Let it run

    The runner executes the real retrieval/answer pipeline per item, then scores it with deterministic checks plus one batched judge call. It records the KB fingerprint at start and end and flags the run kbUnstable if an import landed mid-run.

Read the metrics

A finished run reports overall and per-tag accuracy, groundedness, and citation validity. A run's status is succeeded, degraded (some results errored/timed out — a provider outage reads as degraded, not a regression), or failed. Each result carries an explicit ok / error / timeout status and evidence snapshots of the cited/retrieved chunk text.

A/B diff two runs

Swap a model or a prompt, re-run, then compare at /w/<workspace>/evals/diff. Because a golden set is small, per-item assertion flips lead the diff— you see exactly which claims went from pass to fail — with confidence intervals labeled "insufficient items" below ~30 rather than dressed up as significant. The comparator refuses to compare runs judged by different judge models.

Human override

When the judge is wrong, override its verdict. Overrides are stored as append-only revisions (PATCH /api/w/:workspaceId/evals/judgments/:id), never a destructive edit — the original judgment and who overrode it are both preserved.

Drift

Scheduled re-runs (default off, budget-governed) catch regressions no one triggered, tracking entity counts, orphan rate, and doc churn with consecutive-breach alerting. See Evals & drift.

Endpoints

  • GET/POST /api/w/:workspaceId/golden · GET/PATCH/DELETE …/golden/:id — golden items.
  • GET/POST /api/w/:workspaceId/evals/runs · GET …/evals/runs/:id — eval runs.
  • GET …/evals/diff · GET/POST …/evals/drift · GET/POST …/evals/configs.