Evaluation framework

The first-generation evals model answered one question: did this answer's claims hold up? That is necessary but not sufficient. An answer can be perfectly faithful to the wrong chunks, or right by luck when retrieval missed the source it should have used. The evaluation framework separates the pipeline into layers it can measure independently — retrieval quality, answer quality, and abstention behavior — using the established RAG-evaluation literature, and it does so on a config the runner actually applies rather than merely records.

Three measurable layers

Each family answers a different failure mode. Read them together and you can attribute a regression to its cause instead of guessing:

• Retrieval metrics — did the pipeline surface the right evidence, and rank it well? Covered in depth in Retrieval metrics.
• Answer metrics (RAGAS) — given what it retrieved, is the answer faithful, relevant, and well-supported?
• Abstention & truthfulness (CRAG) — did it answer when it should have, and stay silent when it should not have guessed?

Concept anchoring: what "relevant" means

A golden item no longer anchors only to documents. It anchors to concepts — the entities an answer must be grounded in. The golden_concepts table binds an item to entity ids with a relevance grade: required (the answer is wrong without it), acceptable (helpful, not mandatory), or irrelevant (a near-miss the retriever should not pull). A must/should assertion can name a conceptIdit depends on, so the runner scores "did the pipeline actually detect the entity this claim needs" — not just "did the prose look right." This is the difference between an eval that rewards a lucky paraphrase and one that rewards genuine grounding.

Graded relevance & qrels

Retrieval IR metrics need a relevance judgment per (query, chunk) pair — the qrels of the TREC tradition. The qrels table stores graded judgments (grade 0–3: irrelevant → perfectly relevant) with a source stamp:

• provenance— free and deterministic: a golden item's anchored citations are the highest-grade judgments, derived without a model.
• llm — a graded-relevance judge (UMBRELA-style, below) labels the rest of the retrieved pool, stamping its judge_model and prompt_version. Verdicts are cached back into qrels so a re-run is free and reproducible.
• human — an override appends a new row, never mutates. The table is append-only; the unique key is (query, chunk, source, judge_model) so a human correction and the LLM judgment it disputes both survive, auditable side by side.

Answer metrics: RAGAS, ported

The answer layer ports the RAGASmetric definitions onto the platform's existing batched judge (the same judgeAssertions engine that powers grounded chat — reused, never forked), each storing its judge_model and prompt_version:

• Faithfulness— the fraction of the answer's claims entailed by the retrieved context. The direct measure of hallucination.
• Response relevancy — how well the answer addresses the actual question (RAGAS estimates this by generating questions the answer would answer and comparing them to the original).
• Context precision — are the relevant chunks ranked at the top of what was retrieved? A precision-at-rank signal on the context itself.
• Context recall — did retrieval surface everything the reference answer needed?
• Noise sensitivity — how often an irrelevant retrieved chunk corrupts a claim. RAGAS added this precisely because a high-faithfulness answer can still be led astray by a plausible distractor.

Abstention & CRAG truthfulness

A RAG system that never abstains will confidently answer questions its corpus cannot support. The CRAG benchmark formalized the scoring that makes silence measurable — and makes a wrong answer cost more than no answer:

perfect     → +1.0    fully correct, well grounded
acceptable  → +0.5    correct but minor omission / imprecision
missing     →  0.0    an honest abstention ("I don't know")
incorrect   → −1.0    a hallucination or wrong claim

truthfulness ∈ [−1, 1]   — a hallucination is scored strictly worse than an abstention

The runner designates each result abstained or not, computes truthfulness via one judge call, and pairs it with a risk–coverage curve and its summary AURC (area under the risk–coverage curve) over {confidence, correct} pairs — the same selective-prediction lens the answer policy uses to pick its floors, now measured on the golden set. A workspace can read exactly how much coverage it buys per point of risk before it trusts the pipeline to abstain in production.

Golden items carry the CRAG facets too — answerable (is the corpus expected to support this at all?), questionType (simple · conditional · set · comparison · aggregation · multi_hop · post_processing · false_premise), popularity (head/torso/tail), and dynamism (static/slow/fast) — so metrics break down by question difficulty and by how fast the truth moves, not just by business unit. A false_premise item is a trap: the correct behavior is to reject the premise, and abstention scoring is what lets you reward that.

Error classification

Every result is labeled with one errorClass, turning a bag of failures into a diagnosable distribution written to the append-only eval_feedback queue:

• retrieval_miss — the evidence was never retrieved. Fix the corpus or the retriever, not the prompt.
• hallucination — a claim with no support in what was retrieved.
• citation_error — right claim, wrong or missing [Sn] marker.
• stale — grounded in an out-of-date source (the dynamism facet is what makes this detectable).
• incomplete — a must-assertion missing, though what was said is correct.
• conflict_mishandled — sources disagreed and the answer picked one silently instead of surfacing both.
• correct — the happy path.

Because the queue is append-only and content corrections travel with it, a classified failure can be promoted back into a golden item — the loop that turns a caught regression into a permanent test.

Promote a reviewed correction

A reviewer can turn a corrected eval result into a regression test with POST /api/w/:workspaceId/evals/feedback/promote. This path takes assertions the reviewer has already corrected; it does not call a model. It first verifies that the run belongs to the workspace and, when supplied, that resultId belongs to that run. It then uses one workspace transaction to create the draft golden item and its must-assertions, citation anchors, and tags, while appending the linked eval_feedback audit row. A validation or write failure rolls back both sides.

curl -X POST https://your-host/api/w/$WORKSPACE_ID/evals/feedback/promote \
  -H "Cookie: $SESSION_COOKIE" \
  -H "Content-Type: application/json" \
  -d '{
    "runId": "01KZRUN0000000000000000000",
    "resultId": "01KZRESULT0000000000000000",
    "question": "What is the approved retention period?",
    "correctedAssertions": ["Audit exports are retained for seven years."],
    "citations": [{"docId": "01KZDOC0000000000000000000", "mode": "required"}],
    "errorClass": "incomplete",
    "tags": ["topic:retention"]
  }'

{
  "item": {
    "id": "01KZGOLDEN0000000000000000",
    "status": "draft",
    "question": "What is the approved retention period?",
    "assertions": [
      { "text": "Audit exports are retained for seven years.", "kind": "must", "weight": 1 }
    ],
    "citations": [
      { "docId": "01KZDOC0000000000000000000", "mode": "required" }
    ],
    "tags": ["topic:retention"]
  },
  "feedback": {
    "id": "01KZFEEDBACK00000000000000",
    "runId": "01KZRUN0000000000000000000",
    "resultId": "01KZRESULT0000000000000000",
    "errorClass": "incomplete",
    "correctedBy": "user_01KZ",
    "note": "Correction promoted to golden item 01KZGOLDEN0000000000000000."
  }
}

Component vs. system

The runner can execute in two modes. System mode runs the full answer pipeline and scores the end product. Component mode runs retrieval only, scoring recall/nDCG against the qrels without spending generation tokens — so you can tune the retriever in a fast, cheap loop and only run the expensive full-system eval once retrieval is where you want it. It is the same distinction a search team draws between offline IR evaluation and end-to-end evaluation, made a first turn of the dial rather than a fork of the harness.

Significance: two gates, seeded

A golden set is small, so a raw "+3 points" means nothing without a significance test. Comparison runs a paired bootstrap with BCa (bias-corrected and accelerated) confidence intervals and a paired permutation test, both seeded and deterministic (a mulberry32 PRNG — the same seedable generator the confidence model uses), and it only calls a difference real when both gates pass: the CI excludes 0 and p < 0.05. Two independent tests agreeing is a deliberately conservative bar — it is designed to refuse to cry "improvement" on noise, the single most common way an eval product lies to its owner.

Research grounding

None of these metrics is invented here — each maps to a published benchmark or an established IR definition, which is what lets the numbers be compared to the outside world: