Crucible

Build golden sets

A golden set is the ruler the evaluation framework measures against. This guide builds one that scores retrieval and abstention, not just answer prose: concept-anchored items with CRAG difficulty facets, graded relevance judgments (qrels), and a feedback loop that promotes real failures into permanent tests. It extends the basics in Golden sets & eval runs.

Prerequisites

A key with kb:write (author items and qrels) and kb:read or runs:read (read runs back), exported as $ELI_KEY. See Getting started.
A workspace with documents ingested and extraction run, so there are entities to anchor concepts to.

1 · Write concept-anchored items

Question, must/should assertions, and the concept each needs
Start as always — a question plus atomic must / should assertions. What is new: an assertion can name the conceptId(an entity id) it depends on. "The FY26 budget freezes Operations headcount" requires the FY26 Budget entity to have been detected; anchoring it there means the runner scores genuine grounding, not a lucky paraphrase.
Grade the concepts
On the item, attach golden_concepts: entity ids with a relevance grade — required (wrong without it), acceptable (helpful), or irrelevant (a near-miss the retriever should not pull, e.g. a same-named entity from another business unit).
Set the CRAG facets
Tag the difficulty and volatility so metrics break down by them: answerable (is the corpus expected to support this?), questionType (simple · comparison · aggregation · multi_hop · false_premise · …), popularity (head/torso/tail), and dynamism (static/slow/fast).

Include unanswerable and false-premise items on purpose

A golden set of only answerable questions can never measure abstention. Add a handful with answerable: false and at least one false_premiseitem ("When did we sunset the Atlas product?" when Atlas was never sunset). The correct behavior is to abstain or reject the premise — and CRAG truthfulness scoring rewards exactly that, scoring a confident wrong answer at −1 against an honest missing at 0.

2 · Seed qrels from provenance (free)

Retrieval metrics need a relevance grade per (query, chunk) pair. The cheapest source is provenance: a golden item's anchored citations are, by definition, the highest-grade relevant chunks — no model needed. The runner can author these automatically from the item's citations, or you can post them explicitly:

POST/api/v1/qrelsBearer · kb:write

Author graded relevance judgments for a golden item. Append-only: a new judgment for the same (query, chunk, source, judge) inserts a row; a human override never mutates an existing one. source=provenance needs no judge model.

Request body

`queryId`required	string	The golden item id these judgments belong to.
`grade`required	integer (0..3)	0 irrelevant · 1 marginal · 2 relevant · 3 perfectly relevant.
`chunkId`	string	The chunk being graded (or docId for doc-level judgments).
`docId`	string	The document the chunk belongs to.
`source`required	string (llm \| human \| provenance)	How this grade was produced. provenance = derived from anchored citations, deterministic.

Example requestbash

curl -s -X POST https://your-host/api/v1/qrels \
  -H "Authorization: Bearer $ELI_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "queryId": "01KZGOLDEN000000000000000",
    "chunkId": "01KY10BBBBBBBBBBBBBBBBBBBB",
    "docId": "01KY10AAAAAAAAAAAAAAAAAAAA",
    "grade": 3,
    "source": "provenance"
  }'

Responsejson

{ "id": "01KZQREL0000000000000000A", "queryId": "01KZGOLDEN000000000000000", "grade": 3, "source": "provenance", "createdAt": "2026-07-21T09:14:05.210Z" }

3 · Top up with the graded-relevance judge

Provenance only grades the chunks you already know are relevant. To score precision and nDCG you also need grades for the chunks retrieval returned that were not anchored — the potential distractors. The runner fills these with the UMBRELA-style graded judge (one prompt per pair, 0–3), caching each verdict back to qrels with source: llm and its judge_model + prompt_version. Because verdicts are cached, a re-run is free and reproducible.

A human grade appends — it never overwrites

Disagree with a judge's grade? Post a source: human judgment for the same pair. It inserts a new row; the table is append-only and the unique key includes the source and judge model, so the human correction and the LLM grade it disputes both survive, auditable side by side. The runner prefers human grades when both exist.

4 · Promote failures back into the set

The most valuable golden items come from real misses. When a run classifies a result — a retrieval_miss, a hallucination, a conflict_mishandled — that classification lands in the append-only eval_feedbackqueue with room for a content correction. From there you promote it: the corrected answer becomes a new golden item's assertions, its cited docs become provenance qrels, and the question that broke the pipeline is now a permanent regression test. This closes the loop the feedback model opened — a caught failure becomes a test that stays caught.

the golden-set looptext

run → classify failures (eval_feedback) → correct content → promote to golden item
                                                            ↘ provenance qrels
re-run → the same failure is now a scored, permanent test

5 · Confirm the set is ready

A golden set worth trusting has, roughly:

A spread of questionType and popularity facets — head questions alone flatter the pipeline.
Some answerable: false and a false_premise item, so abstention is measurable.
Provenance qrels on every answerable item, plus judge-graded qrels covering the retrieved pool.
Concept anchors on the assertions that depend on a specific entity.
Enough items that significance is meaningful — small sets get confidence intervals labeled "insufficient items" rather than dressed up as real.

Now measure

With the set built, run it and read recall/nDCG/MRR in Retrieval metrics, then A/B two configs with significance in Measure & compare.