Skip to documentation
eli.aidocs

Crucible

Silent QA sampling

Extraction auto-applies thousands of entity links and relations that no human will ever look at — which means their error rate is invisible by default. Silent QA sampling routes a deterministic ~2% sampleof auto-applied decisions into a human audit queue, turning "we think extraction is good" into a measured precision number per (task, model).

What gets sampled

The pipeline distinguishes decisions a human already reviews from decisions applied silently. Sampling targets only the silent path:

  • Auto-applied entity links — a mention resolved to an existing entity above the confidence threshold, applied without review.
  • Auto-applied relations — an extracted relationship written to the graph above threshold.

Low-confidence decisions already flow to entity review; sampling exists precisely for the high-confidence decisions that skip it.

Deterministic hash-based selection

Whether a decision is sampled is a pure function of its identity: hash the decision's stable id and select when the hash falls under the sample rate (~2% by default, tunable per workspace in workspace settings). No randomness at apply time means:

  • Reproducible — re-running extraction over the same input selects the same sample; the audit set is stable across retries and replays.
  • Unbiased— the hash is uncorrelated with content, so the sample is not skewed toward easy or hard cases the way "review what looks suspicious" sampling is.
  • Un-gameable — a model (or prompt-injected content) cannot steer its own outputs out of the audit sample, because selection depends on nothing the model produces.
selection ruletext
sampled(decision) := sha256(decision.id)[0..4] / 2^32 < rate
# rate = 0.02 (2%) by default — per-workspace setting (qaSampleRate)

The audit queue

Sampled decisions land in a dedicated audit queue — separate from entity review, because the question is different. Review asks "should this be applied?" before the fact; audit asks "was this correct?" after it. The auditor sees the decision with its source context and marks it correct or incorrect; an incorrect verdict also queues the underlying link or relation for fixing. It is silent QA: sampling never blocks or delays the pipeline — decisions apply immediately and are audited after the fact.

Measured precision per (task, model)

Verdicts aggregate into precision — the fraction of audited decisions judged correct — bucketed by (task, model): entity linking on the extraction slot's current model is one series, relation extraction on that model is another. When a model slot changes, a new series starts rather than polluting the old one, so a model swap's quality impact is visible within days.

The number enterprise buyers ask for

"How accurate is the automated extraction?" is the first question a compliance or procurement review asks about an AI-maintained knowledge base. Most vendors answer with a benchmark score from someone else's data. Silent QA sampling answers with measured precision on this workspace's own data, under the models actually configured — an auditable number backed by human verdicts, continuously refreshed.

The measured series also feeds the safety net: a precision drop past the workspace's drift threshold raises an alert through the standard alerts pipeline.

Related