Lens

Semantic caching

Two people ask "what's our refund window?" an hour apart. The pipeline retrieves the same chunks, spends the same generation tokens, and produces the same grounded answer twice. A semantic cache serves the second one from a store — but naïvely, a cache is a correctness hazard: it can serve a stale answer after the source changed, or leak one tenant's answer to another. The design here treats the cache as an opt-in optimization that must be invisible until proven safe: it defaults off, has a shadow mode that measures hit rate without ever serving, tracks exactly which documents each answer depends on, and re-checks readability at serve time.

A cache HIT in shadow is never served — same discipline as policy shadow

cache.mode has three values. off bypasses the cache entirely (zero overhead — not even a lookup). shadow computes hit/miss and logs it but always runs the real pipeline and serves the fresh answer — so hit rate is measurable while behavior is byte-identical to off. enforce is the only mode that serves from cache. This mirrors the answer-policy shadow invariant: you measure the change before you let it touch a single answer.

The tiered lookup

Lookup is cheapest-tier-first. Only the grounded-answer path is cached — structured query and the chat final answer — never agent intermediate steps, where a stale tool result would be dangerous.

Exact-key first (free), then pgvector cosine, then a paid LLM equivalence check in the uncertainty band. Every path that would serve passes a serve-time ACL re-check first; a failed re-check is a miss, never a leak.

Exact tier — exact_key = sha256(normalized question | model | promptVersion | effectiveConfig). A hit is a byte-for-byte identical request; no embedding needed. This tier works even when embeddings are unconfigured (the embedding column is nullable), so the cache degrades to exact-match-only rather than failing.
Semantic direct-serve — a pgvector cosine search over stored question embeddings. A neighbor at similarity ≥ directServeThreshold (default 0.97) is served directly: at that distance the questions are effectively the same.
LLM equivalence tier — in the band [judgeVerifyThreshold, directServeThreshold) (default [0.90, 0.97)) the questions are close but not obviously identical. One cheap judge-slot call decides semantic equivalence; if it says yes, the answer is served and promoted (a new exact-key row is written so the next identical hit skips the judge). Below judgeVerifyThreshold it is a plain miss.

Why a similarity threshold alone is not enough

The uncertainty band is the crux. "How do I cancel my plan?" and "How do I upgrade my plan?" can embed close, but they must not share an answer. A raw cosine cache serves the wrong one; the equivalence tier is the guard that a pure vector store lacks. The two thresholds let a workspace tune the speed/safety trade-off: raise directServeThreshold toward 1.0 to trust only near-identical questions, or lower judgeVerifyThreshold to let the judge adjudicate more of the long tail.

Native pgvector, not a bolt-on store

The embedding is a fixed vector(1536) column on semantic_cache_entries with an HNSW index under vector_cosine_ops — the same Postgres + pgvector the knowledge base already runs on. There is no second datastore to operate, back up, or keep tenant-isolated: the cache lives under the same FORCE-RLS tenancy as every other tenant table, so a cross-tenant read is impossible at the database layer, not just in application code. When embeddings are not configured, the column is null and only the exact tier runs — the cache never hard-depends on an embedding provider.

Dependency-tracked invalidation

A cached answer is only valid while the documents that grounded it are unchanged. On write, the cache records a semantic_cache_dependencies row per [Sn] source in the answer — the doc_id and the document's content_hash at the moment it was cached. When any document changes or is deleted — from vault edits, the API, or a connector sync — ingestion calls cacheInvalidateOnDocChange(workspaceId, docId, newHash), which enqueues a cacheInvalidate job:

Any document write funnels through one hook. The worker looks up dependent entries by doc_id and tombstones those whose stored content hash no longer matches — stamping an invalidation_reason. Entries are never silently deleted; the reason is auditable.

Invalidation compares hashes, not timestamps: an edit that does not change a document's content (a re-sync that yields the same bytes) leaves dependent cache entries intact, so a connector polling every few minutes does not needlessly cold-start the cache. Tombstoning appends an invalidation_reason rather than hard-deleting, keeping the record of why an answer stopped being served.

Serve-time safety

Invalidation is eventual (it rides a queue), so it is a best-effort freshness signal, not the security boundary. The boundary is enforced at serve time, every time:

Tenant scoping — the entry, its embedding search, and its dependencies are all read under FORCE-RLS. A cache row from another workspace is not visible, full stop.
ACL re-check — before serving, every dependency document must be currently readable in the workspace. A document that was deleted, tombstoned, or moved out of visibility since the answer was cached turns the hit into a miss — the pipeline re-runs and the answer reflects what the asker can actually see. A cache never widens access.
Model & prompt in the key — embedding_model is part of the uniqueness key and the model id + prompt version are part of the exact key, so a model swap or prompt change starts a fresh cache generation rather than serving answers a different model produced.
Per-answer TTL — ttlHours (default 168, one week) bounds staleness even for a document that never triggered an invalidation, catching drift that lives outside any single document's hash.

Why the re-check is not optional

Published work on cache-based attacks against LLM semantic caches("CacheAttack"-class results) shows two live risks: a timing side channel that leaks whether a similar question was asked before, and cache poisoning that plants an answer for a later victim query. Tenant RLS closes cross-workspace leakage; the serve-time ACL re-check closes intra-workspace privilege escalation (a cached answer grounded in a doc the current asker may no longer read); and equivalence-tier promotion is scoped to the asking workspace so no entry an outsider planted can be served. Correctness and security both flow from re-verifying at serve time rather than trusting the store.

What is stored

An entry holds the full StructuredAnswer minus usage — replayable verbatim, including its claims and [Sn] sources — plus the bookkeeping the tiers and invalidation need:

a semantic_cache_entries row (abridged)json

{
  "exactKey": "b1946ac9…",            // sha256(question | model | promptVersion | effectiveConfig)
  "question": "what is our refund window?",
  "embedding": "[0.013, -0.041, …]",  // vector(1536), or null when embeddings unconfigured
  "embeddingModel": "text-embedding-3-small",
  "answer": { "answer": "…", "claims": [ … ], "sources": [ { "id": "S1", "docId": "01KY…" } ] },
  "modelId": "claude-sonnet-4-5",
  "promptVersion": "structured-v3",
  "hitCount": 7,
  "lastHitAt": "2026-07-21T09:14:05.001Z",
  "expiresAt": "2026-07-28T09:00:00.000Z",
  "invalidatedAt": null,
  "invalidationReason": null
}

hit_count and last_hit_at are the only fields updated in place; everything else is write-once, and invalidation appends a reason rather than mutating the answer. Hit rate, entry counts, and invalidation totals surface in GET /api/v1/cache/stats and the workspace cache tile.

Research grounding

Mechanism	Where it lives	Research grounding
Tiered exact + semantic lookup	cache/semantic-cache.ts	GPTCache (Zilliz, MIT) pioneered LLM semantic caching; its development is largely frozen, so the design pattern is borrowed and reimplemented natively rather than depended on.
Native pgvector HNSW cosine	semantic_cache_entries.embedding	pgvector HNSW vector_cosine_ops — reuse the KB's own vector store; no separate cache datastore to isolate or operate.
LLM equivalence tier in the uncertainty band	cache/semantic-cache.ts (judge slot)	Similarity alone conflates near-but-distinct questions (cancel vs. upgrade); a verification step is the known fix for semantic-cache false hits.
Serve-time ACL re-check + tenant RLS	answerStructured integration	CacheAttack-class results on LLM semantic caches — timing side channels and cache poisoning; re-verify readability at serve time, never trust the store.
Dependency-hash invalidation (append reason)	semantic_cache_dependencies + processCacheInvalidation	Cache-invalidation-on-source-change; hash comparison (not timestamps) avoids needless cold-starts under frequent connector polling.
off / shadow / enforce	cache settings bag	Same shadow discipline as the answer policy — measure hit rate with zero behavior change before serving a single answer.

Turn it on safely

Start in shadow, watch the hit rate in GET /api/v1/cache/stats for a week, confirm invalidations track your document churn, then flip to enforce. The settings bag lives in workspace settings; the answer shape it replays is structured query.