Crucible

Crucible is the measurement system: golden sets of questions with atomic assertions, graded relevance judgments over retrieved evidence, evaluation runs scored against both, and drift snapshots that notice when the corpus underneath moves. It turns "this feels better" into a number, and then into a signed delta with a confidence interval and a p-value, so a configuration change can be accepted or rejected on evidence. Its public API surface is deliberately small and read-mostly: read one run's aggregated metrics, compare two runs with paired significance, and append a relevance judgment.

Use it standalone

Adopting Crucible alone means one package subpath and one API key. Nothing here imports another capability module, and none of the three endpoints requires a document, graph, connector, or agent to be configured — they read rows that already exist.

React components

Four components are exported from @eli-ai/react/quality, together with the EliQualityLabels, EliQualityProps, and EliPolicyDecisionsProps types. They are client components (the file carries "use client") and each builds only a quality client from the provider's transport. Markup uses scoped eli-* classes; import @eli-ai/react/styles.css for the shipped styling, or leave it out and write your own.

EliProvider takes a transport (or an aggregate client, whose transport is retained) plus an optional workspaceId that the components pass as a per-call override. It has no API-key prop by design.

EliEvalMetrics

The focused Crucible surface. Given a run id it fetches GET /api/v1/evals/runs/{id}/metrics once and renders: a status badge (succeeded positive, failed critical, anything else warning), a resultsCounted/samplesN samples counted line, a KPI grid with a meter per value, abstention badges, and — when the run stored selective-prediction pairs — an inline SVG risk–coverage curve with one point per answered result.

The KPI grid shows exactly seven values, each rendered as a rounded percentage or — when null: accuracy, groundedness, citation precision, citation recall and retrieval hit rate from quality, plus recallAtK and ndcgAt10 from retrieval. The API also returns precisionAtK, mrr, hitAtK, the RAGAS block and the error-class histogram — this component does not draw those, so read them from the endpoint or a hook if you need them.

EliQuality

The composite dashboard. It renders three sections in one surface: an answer-policy table (verdict, question, surface, confidence as a rounded percentage) with a total-decisions badge, an evaluation section whose form takes a run id and loads that run's metrics on submit, and a semantic-cache KPI grid. Policy decisions, policy stats and cache stats load on mount; the evaluation panel is idle until you submit, and the submit button is disabled while the input is blank or a request is in flight.

Two of those three sections do not read Crucible endpoints — see how it composes below. A single kb:read key covers all three.

EliPolicyDecisions

A standalone table of answer-policy decisions — verdict badge, question, surface, and a formatted createdAt — with optional surface and verdict filters. It reads the append-only decision log, not evals; it is exported from this subpath because it shares the quality client. Verdict badges are tone-mapped: answer and proceed positive, abstain and abstain_with_pointers critical, everything else warning.

EliCacheStats

A four-tile KPI grid for semantic-cache health: hit rate (with a meter, or — when null), active entries with the embedding-backed count beneath, invalidated entries with the expired count beneath, and the last hit timestamp with the current policy mode (off · shadow · enforce) beneath. It reads GET /api/v1/cache/stats, which is the answer path's cache, not an eval surface.

HTTP API

Three operations. All are workspace-scoped through the bearer key plus an optional X-Workspace-Id header, and every failure is the shared { error, message } envelope — body-validation failures add an issues array. A run that belongs to another workspace is a 404, not a 403.

Field names above are the response contract; the numbers are illustrative. Three details are worth internalising before you build on it:

• status is the run's own status — running, succeeded, failed, or degraded. Metrics are served for a run that is still running; they are simply partial.
• abstention.meanTruthfulness is on the CRAG signed scale (perfect 1, acceptable 0.5, missing/abstained 0, incorrect −1), so it is not a percentage and can be negative. aurc is the mean risk over the swept operating points — lower is better — and is null when the run stored no selective-prediction pairs, in which case riskCoverage is an empty array.
• errorClasses is a histogram keyed by the deterministic error taxonomy the runner assigns per result: retrieval_miss, hallucination, citation_error, stale, incomplete, conflict_mishandled, correct. Results with no error class contribute nothing.

Grades collapse per document at read time with the precedence human > provenance > llm; within the winning source the maximum grade wins. A successful append also writes a qrel.author row to the audit log, targeting the queryId.

Data elements

Every table below carries workspace_id and a *_tenant_isolation row-level-security policy asserting workspace_id = current workspace for both USING and WITH CHECK, so every read and write in the module runs inside a workspace transaction under FORCE RLS. There is no cross-workspace read path.

Two schema decisions shape what a consumer can rely on. Qrel keys are plain text, not foreign keys — queryId and chunkId carry no referential constraint, so judgments survive golden-item edits and chunk churn from re-ingestion. And a run pins what it was graded against: goldenSetHash plus the targetSnapshotId / judgeSnapshotId foreign keys into immutable configuration snapshots mean a historical comparison stays meaningful after the set and the models have both moved on.

Feature map

Five feature areas cover the module: what a workspace measures against, how runs are produced, how per-result rows become numbers, how two runs become a verdict, and how the corpus underneath is watched. Each area names the components the code map and invariants below locate.

System architecture

The write side and the read side are deliberately separate. Runs are produced by the runner — launched from the workspace app or the scheduler — while the three public operations only read what the runner persisted, plus one append-only judgment write. Nothing on the public surface can start or mutate a run.

Where the logic lives

The code map locates each component. Table names stay in the notes; all are defined in the schema module with workspace row-level security.

Primary runtime flow

The highest-value public path is the read-and-judge loop over runs that already exist: aggregate one run, compare it against a baseline with paired significance, and append the human relevance judgments that sharpen the next run's retrieval metrics.

Internals and invariants

Golden sets and graded relevance

Items are revisioned rows with atomic assertions rather than verbatim expected strings; citations anchor to document ids as required or forbidden; tags stratify metrics; concept anchors grade which entities a correct answer must touch. Qrels are plain-text-keyed on purpose so judgments survive item edits and chunk churn, and doc-level grades collapse at read time with source precedence human over provenance over llm, maximum grade within the winning source.

• Invariant — qrels are append-only: a duplicate in the same (source, judge-model) lane no-ops to the existing row, and every successful append writes a qrel.author audit event in the same transaction. Enforced in src/app/api/v1/_lib/evalv2-metrics.ts.
• Invariant— grade resolution applies the human > provenance > llm precedence at read time, never by mutating rows. Enforced by resolveDocGrades in src/server/evals/golden.ts.

Evaluation runs

A run resolves its config, pins the golden-set hash plus immutable target and judge configuration snapshots, and only then executes items. Each item × sample runs the production pipeline, records an evidence snapshot, passes deterministic checks, and has every assertion graded by the judge model. Error classification assigns each result one deterministic taxonomy bucket.

• Invariant — what a run was graded against is frozen: goldenSetHash and the snapshot foreign keys are written before results exist, so later edits to items, configs, or models cannot rewrite history. Enforced in src/server/evals/runner.ts.
• Invariant — provider errors are recorded as errored results and excluded from metric means, and a run whose error fraction crosses the threshold is marked degraded rather than reporting a phantom regression. Enforced in src/server/evals/runner.ts.

Metric math and aggregation

Per-result rows carry the metric payloads; the run level is always derived. The aggregation helper averages retrieval and RAGAS keys over the results that carry them, counts abstentions, sweeps the persisted confidence/correct pairs into the risk–coverage curve and AURC, and reads the overall run_metrics row for the quality block.

• Invariant — a metric mean only includes results that actually carried that metric; nothing contributes zeros, so partial instrumentation cannot dilute a score. Enforced in src/app/api/v1/_lib/evalv2-metrics.ts.

Paired significance

Comparison pairs per-item means across the two runs, then runs a bias-corrected and accelerated bootstrap on the mean paired delta alongside a sign-flip permutation test. One seeded generator drives both resamplers.

• Invariant — identical run pairs and metric always produce identical intervals and p-values: the statistics are pure and seeded, with no unseeded randomness. Enforced in src/server/evals/metrics/significance.ts.

Drift, freshness, and alerts

Snapshots fingerprint the corpus — entity counts per type, orphan rate, and document churn since the previous snapshot. The scheduled scan evaluates thresholds from workspace settings and raises one deduped alert per open condition; staleness compares the newest document and snapshot ages against the configured window.

• Invariant — alert rows commit in a short transaction and webhook fan-out runs only after the commit, so a delivery failure can never lose or duplicate the alert itself. Enforced in src/server/evals/drift.ts.

For a self-hosting team the additive seams are the safe ones: contribute run metrics through the plugin surface (namespaced into the run_metrics plugin bag, so first-party numbers are never overwritten), author qrels from your own review tooling over the public endpoint, and tune drift thresholds per workspace. The pinning discipline — golden-set hash, config snapshots, seeded statistics — is what makes historical comparisons meaningful and is not designed to be relaxed.

How it composes

Crucible is usable on its own for reading and judging, but it measures something — and that something is produced by other modules. Here is the honest split.

• Lens — needed to produce runs. An eval run routes each golden item through the real retrieve-and-answer pipeline; that is where retrievalMetrics, abstained and truthfulness come from. Without an answering pipeline there is nothing to score. Reading metrics and comparing runs still work perfectly on runs that already exist.
• Warrant — powers two of EliQuality's three sections. EliQuality and EliPolicyDecisions read GET /api/v1/policy/decisions and GET /api/v1/policy/stats — the append-only answer-policy decision log written by the chat, structured and agent pipelines. Without the policy engine those panels render their empty state; the evaluation section is unaffected. Same kb:read scope, different system.
• Atlas — unlocks concept-anchored scoring. golden_concepts.entityId and golden_assertions.conceptId reference graph entities, and the drift snapshot's entity counts and orphan rate are computed from entities and their mentions. With no graph, leave conceptId null — assertions stay plain text and everything else still scores.
• Intake — gives citations and drift something to point at. Golden citations anchor to document ids, and source: "provenance"qrels are derived from an item's required citations (graded 3, keyed at document granularity). Drift's docChurn and docCount are counts of ingested documents. Human and LLM qrels need none of this.
• Ports — how the three operations are published. getEvalRunMetrics, compareEvalRuns and authorQrel are catalogued under the quality capability, with kb:read / kb:write as the canonical scopes and runs:read retained as an accepted read scope on the two GETs.
• Lineage — complementary, not required. Crucible writes its own audit event for qrel authoring and keeps its own append-only trails (qrels, eval_feedback, judgment overrides). Nothing in the module reads a lineage surface.

Conduit has no direct relationship with this module: nothing in Crucible's code reads a data connector or a live query. If an evaluated answer happens to include live data, that arrives through the answer pipeline, and Crucible sees only the resulting text and citations.

For the conceptual model behind the scores, see the evaluation framework and Evals & drift; for authoring golden items and launching runs in the workspace UI, see Golden sets & eval runs and Retrieval metrics.