Lens

Lens is the module that turns a question into an answer you can check. It is one of eight capability modules and it is adoptable on its own: two React components, two primary HTTP endpoints, and three read-only observability endpoints.

What it does

Lens searches a workspace's documents and answers questions from them. Every answer comes back with its evidence attached: the atomic claims the model made, the [Sn] sources each claim cites, a retrieval-confidence summary, and an answer-policy verdict. The policy is the part that makes the output trustworthy — before generating it can decide to ask a clarifying question or abstain with pointers to the closest documents, and after generating it attaches a calibrated confidence and caveats rather than rewriting the model's prose.

The raw retrieval channel is also exposed on its own. GET /api/v1/search ranks passages with no model invoked at all, which makes it usable in a workspace that has no chat model configured.

Use it standalone

Adopting Lens alone means one package entry, one API scope, and a workspace that has documents in it. Nothing else in the eight-module surface has to be wired up.

React components

@eli-ai/react/query exports exactly two components. Both render into the package's scoped eli-* classes and set eli-root on the outer element, so the optional @eli-ai/react/styles.css export styles them without leaking into your own CSS. Both are client components and must be rendered inside an EliProvider.

EliQuery

A single surface with two tabs. In search mode it renders a list of ranked passages — document title, path, snippet, a percentage badge and a relevance meter, each row a button that fires onDocumentSelect. In ask mode it renders the answer text, a badge for the policy verdict, a verified/total grounded badge when groundedness was computed, and the list of sources. Errors render an inline retry button. A search that matched nothing renders an empty-state card; ask mode renders a placeholder card until the first answer arrives.

EliSearch

A convenience wrapper that renders EliQuery with allowAsk={false} and initialMode="search" pinned. It accepts every EliQuery prop except those two, so it is the component to reach for when you want retrieval with no model spend and no way for a user to trigger one.

HTTP API

Five routes. The first two are the module; the last three are its read-only observability surface. Every request carries Authorization: Bearer …, plus X-Workspace-Id when the key is a multi-workspace user key. The apex domain 308-redirects, so non-GET curl examples pass -L.

Reading the answer

Four fields on the response are nullable and mean something specific when they are null:

• retrieval is null when nothing was retrieved at all. Otherwise it is the fused summary: topScore, meanScore, chunkCount, and channels, a Record<string, number> keyed by channel name (fts, vector).
• groundedness is null when no judge model resolved or the check failed. It is a two-field shape and nothing more: { verified, total }, both integers — counts of claims the judge could verify against the evidence.
• policy is null when the effective mode is off, or when a gate failed (a policy fault never fails the answer). The verdict is one of six values — proceed, clarify, abstain_with_pointers from the pre-gate; answer, answer_with_caveat, abstain from the post-gate. On a pre-gate short-circuit the block also carries clarify and pointers.
• usage is null only on a served cache hit. A pre-gate short-circuit reports zero tokens rather than null, because no model ran.

retrievalDebug is optional and purely informational — it never changes the sources or the verdict. See Retrieval pipeline for what each stage count means, and Answer policy for how confidence is computed.

Data elements

Every table below is tenant-scoped: each carries a workspace_id column and a row-level-security policy that constrains both reads and writes to the workspace of the surrounding transaction. Lens never selects across workspaces, and a key resolved for workspace A cannot read workspace B's rows even through a crafted id.

Feature map

The surfaces above resolve to six feature areas. Each names the concrete components that deliver it — the same names the code map and the invariants below pin to repository paths — and where a consumer actually meets it.

System architecture

Two entry points share one pipeline. The search route stops after retrieval; the query route continues through the policy gates, the chat model, and the decision log. The cache fronts only the answer path, and the observability routes are pure reads over what the gates and the cache already persisted.

Where the logic lives

The code map pins each component to its module. Table names stay in the notes; all of them are defined in the schema module with workspace row-level security.

Primary runtime flow

The highest-value path is queryWorkspace: one question in, one gated, cited, logged answer out. The pre-gate can finish the request deterministically before any chat-model spend; the post-gate scores whatever the model produced and records both decisions.

Internals and invariants

Hybrid retrieval

The primitive runs its two channels in parallel: a websearch-style full-text pass that retries with a broadened OR rewrite when the strict pass misses entirely, and a pgvector cosine pass that embeds the question through the workspace's active embedding space. Reciprocal-rank fusion (k = 60) merges the lists with a bonus for chunks both channels found, and adjacent chunks of the same document collapse into one result.

• Invariant— deleted documents and rows outside the caller's content ACL are excluded inside the channel queries, before ranking, so nothing downstream ever handles an unreadable chunk. Enforced in src/server/retrieval/index.ts with predicates from src/server/authz/content-acl.ts.

Pipeline orchestration

The orchestrator layers understanding, graph fusion, reranking, and assembly over the primitive. Each stage is wrapped in its own try/catch: a failure skips the stage, records it in the stage counts, and continues. Assembly enforces the token budget with MMR de-duplication for answer context, or returns a plain ranked listing for the search route.

• Invariant — a stage failure never fails retrieval; understanding, graph fusion, and rerank each degrade to a no-op recorded in stages. Enforced in src/server/retrieval/pipeline.ts.
• Invariant — with all stages inactive the pipeline returns the raw retrieve result byte-identical, chunk set and order, which is what keeps eval baselines reproducible. Enforced in src/server/retrieval/pipeline.ts.

Grounded answer generation

The orchestrator resolves the chat model, assembles the [Sn] context, and requests one generation whose claims must cite their sources. Entity detection explains which graph nodes the question touched; when live data is requested, the data bridge executes fully-resolvable bindings and the answer gains [Dn] rows. Groundedness runs after generation as a best-effort judge pass.

• Invariant — model and provider calls happen outside any workspace database transaction; retrieval and detection use their own short read transactions. Enforced across src/server/query/structured.ts and src/server/retrieval/pipeline.ts.

Answer policy

The pre-gate orders its checks deterministic-first: empty or below-floor retrieval abstains with pointers, a name resolving to multiple live entities clarifies with options, and only then may an optional single LLM pre-check run. The post-gate is pure: it scores the calibrated logistic confidence over the named feature vector and routes to answer, answer_with_caveat, or abstain.

• Invariant — every gate evaluation appends one decision row, in shadow mode as in enforce, and a policy fault degrades to policy null rather than failing the answer. Enforced by src/server/policy/decisions.tsand the orchestrator's gate wiring.
• Invariant — an answer resting only on unverified sources has its confidence hard-capped below the answer floor, so it can never pass as a clean answer. Enforced by postGate in src/server/policy/router.ts.

Semantic answer cache

Lookup proceeds exact key, then cosine similarity above the direct-serve threshold, then a cheap LLM equivalence check for the band below it. A candidate only serves after revalidation: every dependency document must still be readable under the caller's ACL and its content hash must match the hash captured at store time.

• Invariant — a content change or a permission revocation forces a miss at serve time even before asynchronous invalidation catches up. Enforced in src/server/cache/semantic-cache.ts.
• Invariant — fuzzy tiers are scoped by the model, prompt version, and effective config baked into the scope key, so a paraphrase never matches an entry produced under a different configuration. Enforced by src/server/cache/keys.ts.

Access and governance filtering

Authorization is a retrieval concern, not a rendering concern: the ACL predicates join into the channel SQL itself, entity governance labels drop draft and pending-review concepts from answers, and document verification feeds the confidence features rather than being displayed only.

For a self-hosting team the supported extension points are the pluggable edges: an additional reranker kind or an extra recall channel contributed through the plugin surface (both stay behind the host's ACL filtering and budgets), fitted policy coefficients written through the calibration surface instead of the bootstrap defaults, and custom UI built on the headless query client. The gate ordering, the decision log, and serve-time cache revalidation are the contract the rest of the platform assumes and are not designed to be swapped out.

How it composes

Lens is the module the other seven point at. It works alone, and each neighbor makes it measurably better at something specific.

What it gains

• Intake— the honest dependency. Lens ranks and cites documents; something has to put them there. Intake's connectors also populate acl_visibility and acl_principals, which is what lets restricted documents be retrievable by the right people and invisible to everyone else. Without Intake you can still write documents through any other path — Lens does not care how they arrived.
• Atlas — the entity graph turns on two things. Graph fusion pulls in chunks that are structurally related but textually distant, reported as retrievalDebug.stages.graphAdded. And question-entity detection populates the entities array on the answer. With no graph, both degrade silently to nothing and retrieval proceeds on text alone.
• Warrant— governance is what the policy's confidence is partly made of. Document verification feeds the authority features (an answer resting only on unverified sources is capped), entity lifecycle_status keeps draft and pending-review concepts out of answers entirely, and Warrant's calibration surface writes the policy_calibrations row the gates read. Without it every document is unverified and the gates score against bootstrap coefficients — the policy still runs, just less sharply.
• Conduit — genuinely required for one feature. includeData: true executes the data bindings of detected entities and adds [Dn] citations to the answer. With no connectors configured the call does not fail: the data layer is caught, dataCalls comes back empty, and the doc-grounded answer ships. Live data also has one policy consequence — an abstain_with_pointers pre-verdict is overridden to proceed when data actually ran, because rows can ground an answer that documents could not.
• Lineage — Lens hands you docId and chunkId on every source. Lineage is what turns those ids into a provenance trail and a blast radius. Without it the citation is still a real, resolvable document reference.
• Crucible — the only way to know whether a retrieval or policy change helped. Because the two non-deterministic stages both have deterministic-off paths, an eval run can pin Lens to a fully reproducible configuration. Crucible also owns /api/v1/evals and /api/v1/qrels, which are not part of Lens.
• Ports — the same pipeline is reachable over MCP as the eli_query tool, so an external agent gets identical grounding and the identical policy verdict without going through HTTP.

What still works with none of them

A workspace with documents and a chat model gets the complete Lens contract: fused full-text retrieval, cited claims, a retrieval-confidence summary, groundedness when a judge model is configured, and every policy verdict including clarify and abstain. The parts that need neighbors — graph fusion, live data, verification-weighted confidence, fitted calibration — are the parts that were designed to be absent, and each one is a no-op rather than an error when it is.