Evals & drift

Evals are how you know a change made eli.ai better, not just different. A golden set of questions with atomic expected assertions is run through the real pipeline; deterministic checks and an LLM judge score each answer; per-tag metrics let you compare two configs and catch regressions before they reach a client. Drift detection re-runs on a schedule to catch silent decay.

The feedback loop

Golden items are atomic assertions

A golden item is a question plus a set of must / should assertions and doc-anchored citations — nota verbatim expected string. "The FY26 budget freezes Operations headcount" is one assertion; the model can phrase it any way and still pass. Items carry bu:* and topic:* tags for per-segment metrics, are revisioned, and pin a golden-set hash on every run.

Two authoring paths:

• Manual editor — write the question, assertions, and required citations by hand.
• Promote from an answer — the judge model decomposes a real chat/agent answer into candidate atomic assertions; a human edits and approves before it is saved. Never auto-saved.

The runner

The runner executes the real retrieval/answer pipeline per item, then scores it. It is built for honesty at consulting scale:

• Deterministic checks— citation validity, forbidden citations, regex, schema, and retrieval-hit-rate (which separates "the KB lacks it" from "generation failed").
• One batched LLM judge call per result at temperature 0, returning a per-assertion verdict. The judge slot is pinned and snapshotted separately from the target config, and the comparator refuses to compare runs judged by different judges.
• Explicit result status — ok / error / timeout. A provider outage reads as a degraded run, not a 35-point regression.
• KB fingerprint at start and end — a run flagged kbUnstable if an import landed mid-eval. A pre-run cost gate asks for explicit confirmation.

Metrics, regression & A/B

Results roll up into run_metrics — accuracy, groundedness, and citation validity, overall and per tag. Because a golden set is small, per-item assertion flips lead the diff UI(honest signal at n = 3) with confidence intervals labeled "insufficient items" below ~30 rather than dressed up as significant. Comparing two runs by config_snapshot_id is an A/B: swap a model or a prompt, re-run the golden set, and read exactly which assertions changed.

Drift detection

Scheduled re-runs (default off, governed by a per-workspace monthly eval budget) catch regressions no one triggered. KB drift v1 tracks the durable, low-false-alarm signals: entity counts per type, orphan rate, and document churn. A dead-man's-switch heartbeat surfaces an "expected vs actual last run" banner so a silently dead scheduler is visible. Alerts require consecutive breaches and CI-based comparison — alert fatigue is the deadliest failure mode an eval product has.

Next

• Golden sets & eval runs — author items, run with a cost estimate, and read the diff.
• Coverage & gap analysis — where eval accuracy overlays the gap matrix.