Answer policy

Grounded retrieval gets an assistant to cite its answers; it does not get it to know when it should not answer. The answer-policy engine adds that judgment: every question passes a pre-generation gate (proceed, ask a clarifying question, or abstain with pointers) and every generated answer passes a post-generation gate (answer plainly, answer with caveats, or abstain) driven by a calibrated hybrid confidence score. The policy never rewrites model prose — it attaches a machine-readable policy block to the answer and, in enforce mode, decides whether generation happens at all. Every verdict is logged to an append-only decision log so thresholds can be tuned from evidence instead of vibes.

The four verdicts

Two verdicts are pre-generation (clarify, abstain_with_pointers — plus proceed, which just means "generate"), and three are post-generation (answer, answer_with_caveat, abstain):

• abstain_with_pointers— never a bare "I don't know." When retrieval is empty, the top fused score is below retrievalScoreFloor, or the pre-gate judge marks every chunk irrelevant, the verdict carries pointers to the closest documents that were found (up to 5, in fused order), so the user always gets somewhere to look next.
• clarify— a question with a genuinely ambiguous referent gets one clarifying question with concrete options instead of a guessed answer. LLMs recognize ambiguity but won't ask unprompted (the CLAMBER benchmark result) — so the router asks for them.
• answer / answer_with_caveat / abstain — thresholded on calibrated confidence: at or above answerConfidenceFloor (default 0.55) answer plainly; at or above caveatConfidenceFloor (default 0.30) answer with explicit caveats; below that, abstain.

The pre-gate: deterministic first

The pre-gate runs cheapest-first, and everything deterministic runs before any model is consulted:

1. Retrieval floor — empty retrieval or a top fused score below the floor is an immediate abstain_with_pointers. No LLM involved.
2. Entity ambiguity, in SQL — if a name or alias mentioned in the question resolves to two or more distinct live, published entities (word-boundary matching over normalized names), the verdict is clarify with options rendered as Name (type). Deterministic, free, and it catches the most common ambiguity class — "which Atlasdo you mean, the service or the team?" — without a judge.
3. One LLM pre-gate call (optional, llmPreGate) — a single structured judge call does three jobs at once: per-chunk relevance labels (the Onyx chunk-relevance filter pattern — all-irrelevant ⇒ abstain with pointers), an ambiguity classification over the CLAMBER-style taxonomy (none · entity · scope · underspecified, with a generated clarifying question), and a disagreement report when two chunks visibly conflict about the same fact.

The disagreement report is not just metadata. When the pre-gate sees sources conflict, the generator is told about it — a hint like "Sources A and B disagree about the refund window — present both accounts with citations" is threaded into the generation prompt. Telling the generator the conflict, rather than hoping it notices, is the single largest lever in the conflict literature (≈ +24 percentage points on conflict-surfacing accuracy in a healthcare-RAG study).

Hybrid confidence

The post-gate's confidence score is a logistic model over 13 named features, drawn from four signal families:

• Retrieval signals — top fused score, top-1→top-2 margin, mean of the top-k, chunk count, fraction above the floor, and channel agreement (the fraction of retrieved documents found by two or more retrieval channels — vector and full-text agreeing is real evidence).
• Verification signals — from concept governance: the best verification rank among cited documents, the fraction verified, whether only unverified documents ground the answer, and whether a detected entity is a certified concept.
• Groundedness — the fraction of the answer's claims the judge verified against the cited evidence (the dominant weight in the bootstrap coefficients).
• Verbalized confidence — as a feature only.The model's self-reported confidence is one input among 13, never the score itself. Measured expected-calibration-error for verbalized RAG confidence exceeds 0.4— a model saying "I'm 90% sure" is close to uninformative on its own, but still carries some signal when weighed against retrieval and groundedness.

Judge-derived features that never ran (no judge configured, check skipped) are imputed at a neutral 0.5 — a missing judge pass neither boosts nor tanks confidence.

Calibration: Platt scaling & risk-coverage

The shipped coefficients are bootstrap-v1 — hand-set, deliberately conservative, and labeled as unfitted by their version string. The intended lifecycle is:

1. Run in shadow or enforce and let the decision log accumulate feature snapshots.
2. Label a few hundred decisions correct/incorrect and refit with Platt scaling — a seeded, fully deterministic logistic fit. Platt-style parametric calibration is the right tool at 200–1k labels (isotonic regression only pays off past ~1k); the same inputs always produce the same coefficients, and each fit mints a new calibrationVersion so every stored confidence is traceable to the coefficients that produced it.
3. Pick thresholds from the risk-coverage curve: for each candidate answer floor, coverage = the fraction of questions answered, risk = the error rate among them. Published RAG selective-prediction results reach ~95% answer precision at ~30% abstention — the curve makes that trade-off explicit for your workspace instead of assuming it. The walkthrough lives in Tune the answer policy. The same risk-coverage and AURC lens — plus CRAG abstention scoring that penalizes a confident wrong answer (−1) below an honest abstention (0) — is measured directly on a golden set by the evaluation framework, so you can validate that the floors you pick actually improve truthfulness before trusting them in production.

Conflict detection

After generation, claims that cite two or more distinct documents are checked for evidence conflicts in one batched judge call. Detected conflicts are typed — direct (flat contradiction), partial (overlapping but inconsistent), temporal(both were true at different times) — and carry both stances verbatim, so a consumer can render "doc A says X, doc B says Y" without re-reading either document. Conflicts are persisted to an append-only evidence_conflicts table (deduplicated per claim/chunk pair, stamped with the detecting model id) and surface as a conflicting_evidence caveat on the answer.

The authority cap

One rule is absolute, borrowed from Glean's verification model: an answer grounded only in unverified documents can never be stated plainly. When no cited document is verified or certified, calibrated confidence is hard-capped just below the answer floor — the best such an answer can do is answer_with_caveat, with an explicit no_verified_source caveat. No feature weight can override it, because it is not a probability statement: it is a policy that unvetted sources never produce unhedged answers. The verification tiers themselves are maintained through concept governance — this cap is where that curation work pays off at answer time.

Modes & rollout: shadow → enforce

The engine has three modes (policy.mode in workspace settings):

• off — the engine does not run.
• shadow — every verdict, confidence, and caveat is computed and logged; behavior never changes. Shadow is the calibration mode: it builds the labeled dataset that makes enforce trustworthy.
• enforce — verdicts gate behavior: clarify and abstain short-circuit generation, caveats and confidence ship on every answer.

{
  "policy": {
    "mode": "enforce",
    "answerConfidenceFloor": 0.55,
    "caveatConfidenceFloor": 0.30,
    "retrievalScoreFloor": 0.005,
    "llmPreGate": true,
    "clarifyEnabled": true,
    "conflictDetection": true
  }
}

Research grounding

Like the governance model, none of this is invented here — each mechanism maps to a published result or a proven production pattern: