Crucible

Measure & compare

You changed something — turned on a reranker, swapped a chat model, edited a prompt. Did it actually help, or did the golden set just wiggle? This guide runs two configs through the evaluation framework, reads the three metric families back, and compares them behind a significance test built to not lie to you.

Prerequisites

A concept-anchored golden set with qrels — build one in Build golden sets.
A key with kb:read or runs:read, exported as $ELI_KEY.

1 · Pin the config you are testing

An eval config's params is the full pipeline config the runner applies — not a label. It is also the canonical hash input, so two runs with the same params are directly comparable. The empty params {} means current behavior (retrieval limit 10, no rerank, policy off), so a baseline never drifts:

evalConfigs.params — the applied pipeline configjson

{
  "retrieval": {
    "rerank": true,          // ← the change under test
    "graphFusion": false,
    "understanding": false,
    "tokenBudget": 3000,
    "candidatePool": 40
  },
  "policyMode": "off",
  "promptVersion": "structured-v3",
  "retrievalLimit": 10,
  "chatModelSlot": "chat"
}

Isolate one variable

A clean A/B changes exactly one field. Keep the baseline config at {} (or your current production shape) and the candidate identical except for rerank: true. If you also bump the model and the prompt, a significant difference tells you nothing about which change caused it.

2 · Run both — cheap first with component mode

If the change is retrieval-only (a reranker, graph fusion, a bigger candidate pool), run in component mode first: the runner scores recall / nDCG / MRR against the qrels without generating answers — fast and nearly free. Only when retrieval improves do you spend generation tokens on a full system-mode run to confirm the answer metrics followed. Run the golden set under each config from the evals workspace (/w/<workspace>/evals), confirming the cost estimate for system runs.

3 · Read a run's metrics

Pull the three families for one run. Retrieval and abstention numbers are deterministic; RAGAS numbers carry the judge model that produced them.

GET/api/v1/evals/runs/{id}/metricsBearer · kb:read or runs:read

Retrieval, RAGAS, and abstention metrics for one run — overall and broken down by question facet. Deterministic metrics are exact; judge-based metrics report the judge model and prompt version they used.

Example requestbash

curl -s https://your-host/api/v1/evals/runs/01KZRUNA00000000000000000/metrics \
  -H "Authorization: Bearer $ELI_KEY"

Responsejson

{
  "runId": "01KZRUNA00000000000000000",
  "configHash": "9f2c…",
  "n": 42,
  "retrieval": { "recallAtK": 0.71, "precisionAtK": 0.44, "ndcgAt10": 0.66, "mrr": 0.74, "hitAtK": 0.90 },
  "ragas": { "faithfulness": 0.88, "responseRelevancy": 0.81, "contextPrecision": 0.62, "contextRecall": 0.70, "noiseSensitivity": 0.14 },
  "abstention": { "truthfulness": 0.58, "abstainRate": 0.12, "aurc": 0.081 },
  "judge": { "model": "claude-haiku-4-5", "promptVersion": "umbrela-v1" }
}

4 · Compare with significance

The comparison is the point. It runs a paired bootstrap with BCa confidence intervals and a paired permutation test, both seeded and deterministic, and only calls a difference real when both gates pass: the CI excludes 0 and p < 0.05.

GET/api/v1/evals/runs/compareBearer · kb:read or runs:read

Paired comparison of two runs over their shared golden items. Reports per-metric deltas, BCa bootstrap CIs, permutation p-values, and a two-gate significant flag. Refuses to compare runs judged by different judge models.

Query parameters

`a`required	string	Baseline run id.
`b`required	string	Candidate run id.

Example requestbash

curl -s "https://your-host/api/v1/evals/runs/compare?a=01KZRUNA00000000000000000&b=01KZRUNB00000000000000000" \
  -H "Authorization: Bearer $ELI_KEY"

Responsejson

{
  "a": "01KZRUNA00000000000000000",
  "b": "01KZRUNB00000000000000000",
  "pairedItems": 42,
  "seed": 1337,
  "metrics": {
    "ndcgAt10":    { "a": 0.66, "b": 0.74, "delta": 0.08, "ci": [0.03, 0.13], "p": 0.004, "significant": true },
    "recallAtK":   { "a": 0.71, "b": 0.73, "delta": 0.02, "ci": [-0.02, 0.06], "p": 0.310, "significant": false },
    "faithfulness":{ "a": 0.88, "b": 0.89, "delta": 0.01, "ci": [-0.03, 0.05], "p": 0.520, "significant": false },
    "truthfulness":{ "a": 0.58, "b": 0.63, "delta": 0.05, "ci": [0.01, 0.09], "p": 0.028, "significant": true }
  }
}

Read this example honestly: the reranker moved nDCG@10 (+0.08, CI excludes 0, p = 0.004) and truthfulness significantly, but recall and faithfulness did not budge — which is exactly what a reranker should do: it reorders the same retrieved set (so recall is flat) to put better evidence first (so nDCG rises), and better ordering let the model abstain less wrongly (truthfulness up). The test told you what changed and gave you a mechanism.

Two gates, on purpose

A single test at p < 0.05is wrong one time in twenty by construction — run enough comparisons and you will "discover" improvements that are noise. Requiring the BCa CI to exclude 0 and the permutation p < 0.05is a deliberately conservative bar that resists false positives. On a small golden set, expect many honest "not significant" results — that is the test working, not failing.

5 · Decide, and lock it in

Significant win on the metric you care about, no significant regression elsewhere → adopt the config.
No significant difference → the change did not help on this set. Either it does not matter, or the set is too small/narrow to see it — grow the golden set before you trust a null.
Mixed (nDCG up, faithfulness down) → the trade-off is now explicit and numeric, which is the whole point; decide with eyes open.

Whatever you adopt, the failing items are worth harvesting: promote classified failures from the run back into the golden set (see Build golden sets) so the next comparison is measured against a stricter ruler.

Where to go next

The metric definitions and the config-applied model: Evaluation framework. The IR metrics in depth: Retrieval metrics. Abstention scoring shares its risk–coverage lens with the answer policy.