Crucible

Crucible is the measurement system: golden sets of questions with atomic assertions, graded relevance judgments over retrieved evidence, evaluation runs scored against both, and drift snapshots that notice when the corpus underneath moves. It turns "this feels better" into a number, and then into a signed delta with a confidence interval and a p-value, so a configuration change can be accepted or rejected on evidence. Its public API surface is deliberately small and read-mostly: read one run's aggregated metrics, compare two runs with paired significance, and append a relevance judgment.

Use it standalone

Adopting Crucible alone means one package subpath and one API key. Nothing here imports another capability module, and none of the three endpoints requires a document, graph, connector, or agent to be configured — they read rows that already exist.

React components

Four components are exported from @eli-ai/react/quality, together with the EliQualityLabels, EliQualityProps, and EliPolicyDecisionsProps types. They are client components (the file carries "use client") and each builds only a quality client from the provider's transport. Markup uses scoped eli-* classes; import @eli-ai/react/styles.css for the shipped styling, or leave it out and write your own.

EliProvider takes a transport (or an aggregate client, whose transport is retained) plus an optional workspaceId that the components pass as a per-call override. It has no API-key prop by design.

EliEvalMetrics

The focused Crucible surface. Given a run id it fetches GET /api/v1/evals/runs/{id}/metrics once and renders: a status badge (succeeded positive, failed critical, anything else warning), a resultsCounted/samplesN samples counted line, a KPI grid with a meter per value, abstention badges, and — when the run stored selective-prediction pairs — an inline SVG risk–coverage curve with one point per answered result.

The KPI grid shows exactly seven values, each rendered as a rounded percentage or — when null: accuracy, groundedness, citation precision, citation recall and retrieval hit rate from quality, plus recallAtK and ndcgAt10 from retrieval. The API also returns precisionAtK, mrr, hitAtK, the RAGAS block and the error-class histogram — this component does not draw those, so read them from the endpoint or a hook if you need them.

EliQuality

The composite dashboard. It renders three sections in one surface: an answer-policy table (verdict, question, surface, confidence as a rounded percentage) with a total-decisions badge, an evaluation section whose form takes a run id and loads that run's metrics on submit, and a semantic-cache KPI grid. Policy decisions, policy stats and cache stats load on mount; the evaluation panel is idle until you submit, and the submit button is disabled while the input is blank or a request is in flight.

Two of those three sections do not read Crucible endpoints — see How it composes below. A single kb:read key covers all three.

EliPolicyDecisions

A standalone table of answer-policy decisions — verdict badge, question, surface, and a formatted createdAt — with optional surface and verdict filters. It reads the append-only decision log, not evals; it is exported from this subpath because it shares the quality client. Verdict badges are tone-mapped: answer and proceed positive, abstain and abstain_with_pointers critical, everything else warning.

EliCacheStats

A four-tile KPI grid for semantic-cache health: hit rate (with a meter, or — when null), active entries with the embedding-backed count beneath, invalidated entries with the expired count beneath, and the last hit timestamp with the current policy mode (off · shadow · enforce) beneath. It reads GET /api/v1/cache/stats, which is the answer path's cache, not an eval surface.

HTTP API

Three operations. All are workspace-scoped through the bearer key plus an optional X-Workspace-Id header, and every failure is the shared { error, message } envelope — body-validation failures add an issues array. A run that belongs to another workspace is a 404, not a 403.

Field names above are the response contract; the numbers are illustrative. Three details are worth internalising before you build on it:

• status is the run's own status — running, succeeded, failed, or degraded. Metrics are served for a run that is still running; they are simply partial.
• abstention.meanTruthfulness is on the CRAG signed scale (perfect 1, acceptable 0.5, missing/abstained 0, incorrect −1), so it is not a percentage and can be negative. aurc is the mean risk over the swept operating points — lower is better — and is null when the run stored no selective-prediction pairs, in which case riskCoverage is an empty array.
• errorClasses is a histogram keyed by the deterministic error taxonomy the runner assigns per result: retrieval_miss, hallucination, citation_error, stale, incomplete, conflict_mishandled, correct. Results with no error class contribute nothing.

Grades collapse per document at read time with the precedence human > provenance > llm; within the winning source the maximum grade wins. A successful append also writes a qrel.author row to the audit log, targeting the queryId.

Data elements

Every table below carries workspace_id and a *_tenant_isolation row-level-security policy asserting workspace_id = current workspace for both USING and WITH CHECK, so every read and write in the module runs inside a workspace transaction under FORCE RLS. There is no cross-workspace read path.

Two schema decisions shape what a consumer can rely on. Qrel keys are plain text, not foreign keys — queryId and chunkId carry no referential constraint, so judgments survive golden-item edits and chunk churn from re-ingestion. And a run pins what it was graded against: goldenSetHash plus the targetSnapshotId / judgeSnapshotId foreign keys into immutable configuration snapshots mean a historical comparison stays meaningful after the set and the models have both moved on.

How it composes

Crucible is usable on its own for reading and judging, but it measures something — and that something is produced by other modules. Here is the honest split.

• Lens — needed to produce runs. An eval run routes each golden item through the real retrieve-and-answer pipeline; that is where retrievalMetrics, abstained and truthfulness come from. Without an answering pipeline there is nothing to score. Reading metrics and comparing runs still work perfectly on runs that already exist.
• Warrant — powers two of EliQuality's three sections. EliQuality and EliPolicyDecisions read GET /api/v1/policy/decisions and GET /api/v1/policy/stats — the append-only answer-policy decision log written by the chat, structured and agent pipelines. Without the policy engine those panels render their empty state; the evaluation section is unaffected. Same kb:read scope, different system.
• Atlas — unlocks concept-anchored scoring. golden_concepts.entityId and golden_assertions.conceptId reference graph entities, and the drift snapshot's entity counts and orphan rate are computed from entities and their mentions. With no graph, leave conceptId null — assertions stay plain text and everything else still scores.
• Intake — gives citations and drift something to point at. Golden citations anchor to document ids, and source: "provenance"qrels are derived from an item's required citations (graded 3, keyed at document granularity). Drift's docChurn and docCount are counts of ingested documents. Human and LLM qrels need none of this.
• Ports — how the three operations are published. getEvalRunMetrics, compareEvalRuns and authorQrel are catalogued under the quality capability, with kb:read / kb:write as the canonical scopes and runs:read retained as an accepted read scope on the two GETs.
• Lineage — complementary, not required. Crucible writes its own audit event for qrel authoring and keeps its own append-only trails (qrels, eval_feedback, judgment overrides). Nothing in the module reads a lineage surface.

Conduit has no direct relationship with this module: nothing in Crucible's code reads a data connector or a live query. If an evaluated answer happens to include live data, that arrives through the answer pipeline, and Crucible sees only the resulting text and citations.

For the conceptual model behind the scores, see the evaluation framework and Evals & drift; for authoring golden items and launching runs in the workspace UI, see Golden sets & eval runs and Retrieval metrics.