Retrieval metrics

Answer quality can hide a broken retriever: a model can paper over a missing source, and a good answer can ride on evidence that only landed by luck. Retrieval metrics score the retrieval pipeline on its own terms— did it find the right chunks, and did it rank them well — using the standard information-retrieval measures, computed as pure, deterministic functions over the run's ranked results and the golden set's graded relevance judgments (qrels). They are the component-mode half of the evaluation framework.

The inputs

Every metric is a pure function of two things: the ranked list the retriever returned (doc or chunk ids, best first) and a qrels map from id to graded relevance. Grades are 0–3 in the TREC tradition — 0 irrelevant, 1 marginal, 2 relevant, 3 perfectly relevant. Binary metrics (recall, precision, hit, MRR) treat any grade ≥ 1 as relevant; graded metrics (nDCG) use the full scale. No model runs at scoring time, so the numbers are reproducible to the last digit.

type Qrels = Map<string, number>;                 // id → grade 0..3

recallAtK(ranked: string[], qrels: Qrels, k: number): number;
precisionAtK(ranked: string[], qrels: Qrels, k: number): number;
hitAtK(ranked: string[], qrels: Qrels, k: number): number;     // 0 or 1
mrr(ranked: string[], qrels: Qrels): number;                    // 1 / rank of first relevant
ndcgAt10(ranked: string[], qrels: Qrels): number;               // graded, exponential gain

The five measures

• recall@k — of all the relevant chunks that exist for this query (per qrels), how many are in the top k? The direct measure of whether retrieval found the evidence. Low recall means the answer never had a chance — a retrieval_miss, not a generation failure.
• precision@k — of the top k the retriever returned, how many are relevant? The measure of noise in the context window — low precision is what feeds the noise_sensitivity failures downstream.
• hit@k — a blunt 0/1: did any relevant chunk make the top k? The floor below which an answer is impossible.
• MRR (mean reciprocal rank) — 1 / (rank of the first relevant result). Rewards putting a relevant chunk early; the difference between rank 1 and rank 5 is the difference between 1.0 and 0.2.
• nDCG@10 — the graded, rank-weighted headline. It rewards placing the most relevant chunks at the top, and it is the metric that uses the full 0–3 scale rather than collapsing it to relevant/not.

nDCG with exponential gain — pinned and documented

There are two conventions for DCG's gain function, and they give different numbers, so the choice is pinned and stated rather than left implicit. This implementation uses the exponential gain:

gain(rel)   = 2^rel − 1              // 0→0, 1→1, 2→3, 3→7  (super-linear in relevance)
discount(i) = 1 / log2(i + 1)        // i is the 1-based rank
DCG@k       = Σ  gain(rel_i) · discount(i)   for i = 1..k
IDCG@k      = DCG@k of the ideal ranking (grades sorted descending)
nDCG@k      = DCG@k / IDCG@k          ∈ [0, 1]   (1.0 = perfect ordering; 0 when no relevant results)

The exponential form 2^rel − 1 is the one used by TREC web tracks and the LETOR learning-to-rank benchmarks (it is the gain LambdaMART optimizes), and it is deliberately super-linear: a grade-3 chunk is worth 7, a grade-1 chunk only 1, so surfacing one perfectly-relevant result matters far more than surfacing several marginal ones. The alternative — linear gain, where the gain is just the grade — under-weights the highly-relevant results this platform cares about most. Both are "nDCG"; reporting which one is the difference between a comparable number and a meaningless one.

A worked example

Suppose the qrels grade three chunks — A=3, B=2, C=1 — and everything else 0, and the retriever returns [B, X, A, C] (X is irrelevant):

ranking  = [B, X, A, C]     grades = [2, 0, 3, 1]
gains    = [2^2−1, 0, 2^3−1, 2^1−1] = [3, 0, 7, 1]
discounts= [1/log2(2), 1/log2(3), 1/log2(4), 1/log2(5)]
         = [1.000, 0.631, 0.500, 0.431]
DCG      = 3·1.000 + 0·0.631 + 7·0.500 + 1·0.431 = 6.931

ideal    = [A, B, C]  grades [3,2,1] → gains [7,3,1]
IDCG     = 7·1.000 + 3·0.631 + 1·0.500 = 9.393

nDCG@4   = 6.931 / 9.393 = 0.738

# and the binary metrics on the same ranking:
recall@4    = 3/3 = 1.00        # all relevant chunks are in the top 4
precision@4 = 3/4 = 0.75        # X is the one miss
hit@4       = 1                 # at least one relevant chunk present
MRR         = 1/1 = 1.00        # B (relevant) is rank 1

The metrics disagree on purpose. Recall says "everything was found"; nDCG's 0.738says "but the ordering left points on the table" — the grade-3 chunk A sat at rank 3 behind a distractor. That gap is exactly what a reranker is meant to close, and this is how you prove it did.

Where they surface

Each eval result carries a retrievalMetrics object — { recallAtK, precisionAtK, ndcgAt10, mrr, hitAtK }— computed against that item's qrels using the config's defaultRetrievalK (default 10). They roll up per run and per question facet (type, popularity, dynamism), read back over GET /api/v1/evals/runs/{id}/metrics, and render in the retrieval-metrics panel of the evals dashboard alongside the AURC / risk–coverage chart. In component mode the runner computes these without generating an answer — a fast, cheap loop for tuning the retriever before spending a single generation token.

{
  "resultId": "01KZ…",
  "retrievalMetrics": {
    "recallAtK": 1.0,
    "precisionAtK": 0.6,
    "ndcgAt10": 0.84,
    "mrr": 1.0,
    "hitAtK": 1
  }
}