Skip to documentation
eli.aidocs

Lens

Policy decisions

Every verdict the answer-policy engine produces — on every surface, in every mode — is recorded as an append-only decision row carrying the exact feature snapshot the verdict was computed from. The log is three things at once: the audit surface for "why did the assistant refuse?", the tuning dataset for calibrating confidence thresholds, and the evidence that shadow mode is safe to promote to enforce.

Anatomy of a decision

One row per gate: a pre row for the pre-generation verdict (proceed / clarify / abstain with pointers) and a post row for the post-generation verdict (answer / answer with caveat / abstain), each tagged with the surface it came from — chat (with its conversation id), structured, or agent (with its run id):

a post-stage decision rowjson
{
  "id": "01KZ5AXW9J3F0Q7B2M8T4R6E1C",
  "surface": "chat",
  "conversationId": "01KZ3FJ2P8V5N1D9S0A7H6G4K2",
  "runId": null,
  "question": "What is our refund SLA for enterprise plans?",
  "stage": "post",
  "verdict": "answer_with_caveat",
  "mode": "shadow",
  "confidence": 0.54,
  "calibrationVersion": "bootstrap-v1",
  "features": {
    "topScore": 0.031, "scoreMargin": 0.004, "meanTopK": 0.022,
    "chunkCount": 8, "countAboveFloor": 6, "channelAgreement": 0.5,
    "maxVerificationRank": 0, "fractionVerified": 0, "onlyUnverified": true,
    "hasCertifiedConcept": false, "groundedFraction": 0.8,
    "verbalizedConfidence": 0.9, "relevantChunkFraction": 0.75
  },
  "reasons": ["confidence_capped_only_unverified", "confidence_in_caveat_band"],
  "caveats": [
    { "kind": "no_verified_source", "detail": "No cited document is verified or certified; treat this answer as provisional." },
    { "kind": "unsupported_claims", "detail": "Only 80% of the answer's claims are supported by the cited evidence.", "claimIndexes": [2] },
    { "kind": "low_confidence", "detail": "Calibrated confidence 0.54 is below the plain-answer floor (0.55)." }
  ],
  "createdAt": "2026-07-21T09:14:05.210Z"
}

This row tells a complete story: the model was 90% sure of itself (verbalizedConfidence), but no cited document was verified, so the Glean cap pinned confidence just below the answer floor and the answer shipped with three explicit caveats. Reading rows like this is how you learn whether your floors are honest.

Append-only, by construction

Decision rows are never updated or deleted — policy_decisions (and its sibling evidence_conflicts) are the audit surface for the answer policy. Every row records the calibrationVersion that produced its confidence, so a recalibration never silently rewrites history; changes to the policy settings are audited through the standard workspace-settings audit trail.

The Policy page

The workspace's Policy page (under Insights) shows the last 30 days at a glance: stat cards of decisions by verdict, and the decision table — time, surface, verdict, confidence, the truncated question, and expandable reasons — filterable by surface and verdict. Agent-surface decisions also appear as a policy row in the run inspector for the run that produced them; chat verdicts surface inline as confidence pills and caveat banners on the messages themselves.

API

GET/api/v1/policy/decisionsBearer · kb:read or runs:read

Newest-first decision listing with optional filters. limit clamps to 1..200 (default 50).

Query parameters

surfacestringFilter: chat | structured | agent.
verdictstringFilter by verdict (pre or post), e.g. abstain, clarify, answer_with_caveat.
limitinteger (1..200)Page size. Default 50.
offsetintegerPagination offset. Default 0.
Example requestbash
curl -s "https://your-host/api/v1/policy/decisions?verdict=abstain&limit=50" \
  -H "Authorization: Bearer $ELI_KEY"
Responsejson
{
  "items": [ { "id": "01KZ5AXW…", "surface": "chat", "stage": "post", "verdict": "abstain", "confidence": 0.21, "features": { "…": "…" }, "reasons": ["confidence_below_caveat_floor"], "createdAt": "2026-07-21T09:14:05.210Z" } ],
  "total": 22,
  "limit": 50,
  "offset": 0
}
GET/api/v1/policy/statsBearer · kb:read or runs:read

Decision counts by verdict and by surface over the last 30 days. Pre-stage rows (proceed/clarify/abstain_with_pointers) and post-stage rows (answer/answer_with_caveat/abstain) both count.

Example requestbash
curl -s https://your-host/api/v1/policy/stats \
  -H "Authorization: Bearer $ELI_KEY"
Responsejson
{
  "since": "2026-06-21T09:00:00.000Z",
  "total": 511,
  "byVerdict": {
    "proceed": 240, "clarify": 12, "abstain_with_pointers": 19,
    "answer": 180, "answer_with_caveat": 45, "abstain": 15
  },
  "bySurface": { "chat": 361, "structured": 128, "agent": 22 }
}

From log to calibration

The feature snapshot on every row is exactly what the confidence model scores — which makes the log a ready-made calibration dataset. The loop:

  1. Collect — run in shadow (or enforce) until a few hundred post-stage rows exist.
  2. Label — mark answered questions correct/incorrect (spot-review, or reuse golden-set judgments).
  3. Refit & rethreshold — Platt-scale new coefficients from the labeled rows and read the answer/caveat floors off the risk-coverage curve.

The full walkthrough, curls included, is in Tune the answer policy.

Related

  • Answer policy — the verdicts, confidence model, and caps that produce these rows.
  • Workspace settings — where the policy bag (mode, floors, toggles) lives.
  • Judge governance — the same discipline applied to the judge that feeds groundedness features.