Tune the answer policy

The answer policy ships with deliberately conservative bootstrap floors — they are placeholders for numbers your data should choose. This guide walks the honest loop: collect decisions in shadow, read them back over the API, label outcomes, pick thresholds from a risk-coverage curve, and only then let the policy gate answers.

Prerequisites

• A key with kb:read or runs:read (reading decisions) and agents:run (spot-check queries), exported as $ELI_KEY. See Getting started.
• Owner/admin on the workspace — policy settings are edited in Settings → Answer policy, which is owner/admin-gated like all workspace settings.

1 · Start in shadow

In Settings → Answer policy, set the mode to shadow. Every question now produces decision rows — verdict, confidence, the full feature snapshot — while behavior stays exactly as before: no clarifying questions, no abstentions, no caveats shown. Only the overridden key is stored on the workspace:

{
  "policy": { "mode": "shadow" }
}

2 · Watch the verdict mix

After a few days of traffic, read the 30-day rollup. The pre/post split matters: proceed vs. clarify vs. abstain_with_pointers tells you what the pre-gate would have blocked; answer vs. answer_with_caveat vs. abstain tells you where confidence is landing against the floors.

curl -s https://your-host/api/v1/policy/stats \
  -H "Authorization: Bearer $ELI_KEY"

{
  "since": "2026-06-21T09:00:00.000Z",
  "total": 511,
  "byVerdict": {
    "proceed": 240, "clarify": 12, "abstain_with_pointers": 19,
    "answer": 180, "answer_with_caveat": 45, "abstain": 15
  },
  "bySurface": { "chat": 361, "structured": 128, "agent": 22 }
}

A quick sanity read: with 240 proceeds producing 180 plain answers, ~25% of answers would ship hedged or not at all under enforce. Whether that is right depends on how many of them deserved hedging — which is what the next two steps establish.

3 · Read the decisions that matter

Pull the rows behind the counts, filtered by verdict. The two review queues that pay off fastest: would-be abstains (was the evidence really too weak, or are the floors too high?) and would-be clarifies (real ambiguity, or an over-eager judge?).

curl -s "https://your-host/api/v1/policy/decisions?verdict=abstain&limit=50" \
  -H "Authorization: Bearer $ELI_KEY"

{
  "items": [
    {
      "id": "01KZ5AXW9J3F0Q7B2M8T4R6E1C",
      "surface": "chat",
      "stage": "post",
      "verdict": "abstain",
      "mode": "shadow",
      "confidence": 0.21,
      "calibrationVersion": "bootstrap-v1",
      "question": "What was the Q3 churn for the EU region?",
      "features": { "topScore": 0.012, "chunkCount": 3, "onlyUnverified": true, "groundedFraction": 0.5 },
      "reasons": ["confidence_below_caveat_floor"],
      "caveats": [],
      "createdAt": "2026-07-21T09:14:05.210Z"
    }
  ],
  "total": 15,
  "limit": 50,
  "offset": 0
}

The features snapshot on each row is the explanation: reasonsnames the rule that fired, and the feature values show why. The same rows are browsable on the workspace's Policy page — see Policy decisions for the full row anatomy.

4 · Label, refit, and read the risk-coverage curve

Label a few hundred answered questions correct/incorrect — spot-review, or reuse golden-set judgments where questions overlap. Workspace operators can inspect the current fit/history and risk-coverage curve with GET /api/w/:workspaceId/policy/calibrations, create an inactive candidate with POST /api/w/:workspaceId/policy/calibrations, then activate the reviewed candidate with POST /api/w/:workspaceId/policy/calibrations/:id/activate. Activating default restores the bootstrap coefficients. These session-authenticated routes require governance:read or governance:write. Underneath, two deterministic utilities in src/server/policy/confidence.ts do the statistics:

• fitPlattCalibration(rows) — refits the confidence coefficients from {features, label} pairs. Seeded and fully deterministic: the same rows always produce the same coefficients, stamped with a new calibrationVersion. Platt-style parametric fitting is the right tool at 200–1k labels; below ~200, keep the bootstrap coefficients and tune floors only.
• riskCoverage(rows) — turns {confidence, correct} pairs into the threshold-selection curve.

[
  { "threshold": 0.35, "coverage": 0.91, "risk": 0.14 },
  { "threshold": 0.48, "coverage": 0.82, "risk": 0.09 },
  { "threshold": 0.61, "coverage": 0.71, "risk": 0.05 },
  { "threshold": 0.74, "coverage": 0.52, "risk": 0.02 }
]

Read each point as: if the answer floor were threshold, we would answer coverage of questions and be wrong on risk of those answers. Pick the lowest threshold whose risk you can accept. In the example, a workspace that tolerates 5% wrong answers sets answerConfidenceFloor ≈ 0.61 and answers 71% of questions plainly; the caveat floor is then placed lower, where risk becomes unacceptable even with a warning label (here perhaps ≈ 0.35). Published RAG selective-prediction results land around 95% precision at ~30% abstention — your curve tells you where your corpus actually sits.

5 · Adjust the floors

Back in Settings → Answer policy, set the floors you chose. Each slider maps to one key of the stored policy bag; inner keys override and clear independently, exactly like chatDefaults:

{
  "policy": {
    "mode": "shadow",
    "answerConfidenceFloor": 0.61,
    "caveatConfidenceFloor": 0.35
  }
}

6 · Spot-check enforce per call

Before flipping the workspace, rehearse enforcement one request at a time: POST /api/v1/query takes an optional policyMode that overrides the workspace mode for that call only.

curl -s -X POST https://your-host/api/v1/query \
  -H "Authorization: Bearer $ELI_KEY" \
  -H "Content-Type: application/json" \
  -d '{"question":"What is the churn definition?", "policyMode":"enforce"}'

{
  "answer": "Which \"churn\" do you mean? …",
  "claims": [],
  "sources": [],
  "policy": {
    "mode": "enforce",
    "verdict": "clarify",
    "confidence": null,
    "calibrationVersion": "bootstrap-v1",
    "reasons": ["ambiguous_entity_mention"],
    "caveats": [],
    "conflicts": [],
    "clarify": { "question": "Which \"churn\" do you mean?", "options": ["Churn (metric)", "Churn (report)"] }
  }
}

Ask questions you know are weakly covered and confirm the short-circuits look right: clarifies offer sensible options, abstains point at genuinely-closest documents, and confident questions still answer cleanly.

7 · Flip to enforce

Set mode to enforce. Verdicts now gate behavior on chat and structured surfaces; decisions keep accumulating, so step 2's stats become your ongoing monitor — a sudden abstain spike after an ingestion change is a signal, not noise. Re-run the loop whenever the corpus shifts meaningfully.