Overview

System diagrams

eli.ai is eight systems that compose. This page shows how they fit together — first the master flow of a document and a question through the whole platform, then a detailed internal architecture diagram for each system, with every node mapped to a real module or table. For the narrative overview, see the capability map; to see the systems in action, seed a demo workspace.

How the eight systems interconnect

A document enters through Intake, becomes the Atlas graph, and is governed by Warrant. A question is answered by Lens, which retrieves over that graph and the chunks, joins live systems of record through Conduit, and gates the answer on the policy engine — then hands it to a consumer through Ports. Crucible measures the answers and steers the next round of curation, and Lineage makes the whole thing traceable end to end. The dashed edges are the feedback loops that keep it honest.

The document pipeline (Intake → Atlas → Warrant), the question pipeline (Lens over the graph + Conduit live data, gated by the answer policy, out through Ports), and the two feedback loops — Crucible regressions steering Warrant curation, and every answer traceable back to its source through Lineage.

Each system, inside

Eight diagrams, one per system — inputs on the way in, the internal stages and stores, and what each hands to the next. Every node names a real module or table.

1 · Intake — Connect & ingest

Confluence, SharePoint, Slack, and markdown-upload sources share one contract. Remote connector requests use the shared address-pinned, redirect-revalidating, bounded HTTP client. A database advisory lock serializes every sync kind for a workspace/source across queues and workers. The crawl awaits each idempotent ingestConnectorDocument upsert before persisting the page checkpoint, so acknowledged progress cannot outrun document/revision state. Unchanged content skips re-chunk/re-embed; a real change appends document_revisions, writes ACL principals, reconciles chunks/FTS, and dispatches embedding/extraction. Manual text import uses saveDocument/importMany and converges on the same derived stores through its caller-owned enrichment path.

Intake serializes each source, ingests every normalized document before advancing its checkpoint, writes revision/ACL metadata, and dispatches searchable chunk enrichment.

Key modules & tables

src/server/connectors/sync.ts — advisory lock, inline drainCrawl ingestion before checkpoint, revisions, prune and permission refresh
src/server/connectors/sources/http.ts + src/server/network — validated, address-pinned connector HTTP with redirect, time and size bounds
src/server/connectors/registry.ts + types.ts — Connector fullLoad / poll / listIds / getPermissions contract
src/server/vault/index.ts + src/server/ingest/bulk.ts — recoverable manual save and text bulk-import path
src/server/ingest/chunking.ts — chunkMarkdown + persistChunks; generated tsvector/GIN index
src/server/ai/embeddings.ts — embedChunks → embedMany into the active embedding space
Tables: connector_sources, sync_runs, documents, document_revisions, chunks, chunk_embeddings, embedding_spaces

Deep dive: Intake →

2 · Atlas — The semantic layer

Chunks enter Atlas through zero-model [[wikilink]] resolution and capped co-occurrence edges. Co-occurrence carries explicit synthetic provenance and replaces its own document pair set idempotently. New model work uses the stately extract-v2 queue: the worker snapshots the exact document, ordered chunks, and ontology, completes every provider call before graph mutation, then locks and revalidates those inputs plus latest-run ownership. One transaction wipes the previous model projection, applies every chunk, rebuilds canonical edges, and completes the run, so the last-good graph remains visible on failure or stale work. Missing/superseded runs are acknowledged; content or ontology changes retry from current state. Document deletion purges graph provenance before chunks in the tombstone transaction, retiring only machine orphans while preserving manual/asserted structure. Readers use entities plus canonical_relations, the maintained serve-time edge projection.

Evidenced deterministic links, atomic ontology-typed extraction, and human curation converge on canonical_relations.

Key modules & tables

src/server/graph/extraction.ts — wikilinks, provider-before-mutation extraction, exact snapshot/owner gate, one-TX replace, delete purge and fact tombstones
src/server/graph/cooccurrence.ts — capped deterministic related_to edges with explicit idempotent provenance
src/server/jobs/index.ts + src/worker/index.ts — stately extract-v2 producer/worker contract and legacy queue drain
src/server/graph/ontology.ts + ontology-admin.ts — vocabulary, constraints, templates and re-extraction
src/server/graph/manual.ts + curation.ts + review.ts — human graph edits, revisions and duplicate merges
src/server/graph/tools.ts + query.ts — entity/graph reads over canonical entities and edges
Tables: entities, aliases, mentions, relations, evidence, canonical_relations, extraction_runs, revisions

Deep dive: Atlas →

3 · Warrant — Govern & certify

Two write paths feed the graph: schema-level and destructive edits are gated through change-requests.ts (propose → owner/steward approve → apply), while every mutation and document verification is stamped into append-only entity_revisions (PROV-O) and documents.verification, escalating each concept up the authority ladder machine_extracted → asserted → curated → certified. At answer time that same governance state flows into the policy engine: router.preGate screens retrieval before generation (score floor, deterministic entity ambiguity, an optional LLM relevance/ambiguity call), and router.postGate scores calibrated confidence from confidence.ts — hard-capping it below the answer floor when only unverified sources are cited and adding caveats from evidence_conflicts. The pre- and post-verdicts (proceed/clarify/abstain and answer/answer_with_caveat/abstain) are logged append-only to policy_decisions, which closes the tuning loop: calibration.ts labels those decisions from reviewer feedback, refits a logistic model, and activates one policy_calibrations row that resolveActiveCoeffs feeds back into every future postGate.

Warrant governs two things at once: the authority and revision history of every concept and document, and the pre/post-gate answer-policy engine whose calibrated confidence — hard-capped for unverified sources — decides answer, caveat, abstain, or clarify.

Key modules & tables

src/server/graph/curation.ts — recordRevision, certifyEntity, markReviewed, setAuthority; authority tiers (machine_extracted→asserted→curated→certified) + lifecycle + review SLAs
src/server/graph/change-requests.ts — proposeChange / approveChangeRequest / rejectChangeRequest (Track A gated review, claim → apply → stamp appliedAt)
src/server/vault/verification.ts — verifyDocument (Glean verified/certified tiers, next_verify_at)
src/server/policy/router.ts — preGate (floor · entity ambiguity · LLM relevance) and postGate (4-verdict engine + Glean confidence cap)
src/server/policy/confidence.ts + calibration.ts — logistic calibrated confidence + feedback→recalibrate→activate tuning loop (resolveActiveCoeffs)
Tables: entity_revisions, concept_change_requests, documents.verification, policy_decisions, policy_calibrations, evidence_conflicts (all workspace-scoped, FORCE-RLS)

Deep dive: Warrant →

4 · Lineage — Provenance & traceability

documentLineage(workspaceId, docId) performs ten staged, tenant-isolated SQL reads (the recent-sync read is conditional) for one live document. It returns origin, verification/review state, the total and newest 50 document revision summaries, the newest 50 document audit events, derived chunk/entity/relation counts, cited-answer count, dependent cache entries, and impacted canonical entities. Entity revision history remains available through the separate entity revisions API and is not embedded in DocumentLineage. REST, MCP, the workspace BFF, and the document UI consume the same bounded payload.

DocumentLineage unifies origin, governance, bounded revision/audit summaries, derivation, and downstream blast radius without a model call.

Key modules & tables

src/server/lineage/index.ts — deterministic 10-stage assembler; no LLM, network, or filesystem I/O
src/server/vault/revisions.ts — append-only document revisions with snapshot/diff/activity
audit_log — bounded document.* event trail
src/server/graph/curation.ts + entity revision endpoint — separate concept history
documents + connector_sources + sync_runs — origin and governance
messages, semantic-cache dependencies, mentions/entities — downstream impact

Deep dive: Lineage →

5 · Conduit — The live-data layer

Conduit is the bridge from the semantic graph to systems of record. When a question touches an entity, resolveBindingsForEntity (bindings.ts) gathers the entity's own bindings plus the ones inherited from its type, buildParams materializes each parameter from the binding's paramMap (entity_name, const, or model-supplied), and executeNamedQuery (executor.ts) loads the matching data_queries domain component and its data_connectors row inside one short RLS transaction, then coerces params against each DataParamSpec. The governed tail runs assertReadOnlySql (single SELECT/WITH only), decrypts the credential at call time, and dispatches to the engine adapter (postgres/mysql/snowflake) which rewrites $1..$n placeholders to the driver bind style and caps the result by maxRows and timeoutMs — all strictly outside any tenant transaction, since connector SQL is network I/O. Every execution appends an append-only data_calls provenance row (executed SQL, params, row count, duration, status), and that row's dataCallId is the anchor behind the [Dn] data citation handed into the grounded answer.

Conduit: a bound entity plus a question resolves through entity_bindings to a governed named query, runs read-only against a system of record via the per-engine adapter, and every call — ok, error, timeout, or blocked — appends a data_calls row that becomes a [Dn] citation.

Key modules & tables

src/server/data/bindings.ts — resolveBindingsForEntity + buildParams: the semantic→data join (entity/type → query + paramMap, entity_name/const/model sources)
src/server/data/executor.ts — executeNamedQuery, assertReadOnlySql (SELECT/WITH-only gate), recordDataCall: the governed gate that always writes provenance
src/server/data/adapters/* — getAdapter, wrapWithRowLimit, rewriteDollarParams; postgres.ts / mysql.ts / snowflake.ts are the only driver-touching modules
data_connectors (table) — AES-256-GCM encrypted creds, maxRows, timeoutMs, enabled, allowRawSql escape hatch
data_queries (table) — named parameterized read-only SQL templates ($1..$n) with DataParamSpec[] param specs (the 'domain components')
entity_bindings + data_calls (tables) — entity/type→query paramMap bindings; append-only [Dn] provenance (sqlText, paramsUsed, rowCount, durationMs, status ok/error/timeout/blocked, executedBy, run/conversation linkage)

Deep dive: Conduit →

6 · Lens — Retrieve & answer

The entry boundary resolves an explicit content-ACL context. Cache keys include that authorization scope and every stored answer must record its complete cited dependency set; serve revalidates authorization and hashes. On a miss, the same actor principals filter FTS/vector candidates and document-derived graph seeds before fusion, reranking, and context assembly. The pre-gate can clarify or abstain without generation; otherwise governed data calls add [Dn] provenance, grounded generation produces [Sn]/[Dn] claims, and groundedness/conflict signals feed the calibrated post-gate. Chat, structured REST, MCP, native agents, and evals propagate the same ACL contract.

Actor-scoped cache and retrieval feed graph fusion, ranking, policy, optional live data, and a cited structured result.

Key modules & tables

src/server/authz/content-acl.ts — principal context and SQL predicates
src/server/query/structured.ts — cache → retrieve → policy → data → generation → policy
src/server/retrieval/index.ts + pipeline.ts + graph-fusion.ts — actor-filtered FTS/vector/graph path
src/server/cache/semantic-cache.ts — ACL-scoped keys, all-or-none dependencies and serve validation
src/server/policy/router.ts + conversations/groundedness.ts — verdict and evidence checks
Tables: cache entries/dependencies, chunks/embeddings, canonical graph, data calls, policy decisions

Deep dive: Lens →

7 · Crucible — Evaluate & measure

runEval pins the active golden set and applies an eval_configs row, then routes each item through the real Lens retrieve-and-answer pipeline. Target and judge snapshot ids are indexed foreign keys to immutable config_snapshots. Results carry deterministic error classes and roll up retrieval, answer-quality, abstention, concept, and significance measures. The two promotion paths remain distinct: /golden/promote accepts a supplied answer payload and does not consume a feedback row; authenticated POST /evals/feedback/promote validates a reviewed correction's run/result ownership, then atomically appends eval_feedback and creates its draft golden item, assertions, citations, and tags. A separate steward action must activate either draft, keeping the loop human-controlled.

Concept-anchored evaluation pins configuration lineage, measures the real Lens pipeline, and promotes reviewed corrections only as draft goldens.

Key modules & tables

src/server/evals/golden.ts — golden items, assertions, citations, concepts, qrels, and draft activation
src/server/evals/runner.ts — applied eval config, Lens execution, results, judgments, metrics, and pinned target/judge snapshots
drizzle/0033_eval_snapshot_fks.sql — indexed target/judge snapshot foreign keys with ON DELETE SET NULL
src/server/evals/metrics/ — retrieval, RAGAS, abstention/AURC, and significance metrics
src/server/evals/feedback-classify.ts + evals/feedback/promote route — deterministic classification and atomic reviewed-correction promotion
src/server/evals/drift.ts + src/server/coverage/index.ts — corpus drift, alerts, and coverage overlays

Deep dive: Crucible →

8 · Ports — Programmatic surfaces

Workspace keys bind one workspace; user keys resolve a live membership and capability set, with X-Workspace-Id when required. Both feed REST and the stateless eight-tool/two-resource MCP server under FORCE-RLS and an actor content-ACL context. Agent starts return 202 after an admitted queued outbox row; a stable agent-run identity plus boot/minute reconciliation repairs an interrupted broker handoff, and clients reattach through DB-backed SSE. HITL decisions atomically move suspended runs back to queued and the agent-resume worker idempotently claims the persisted cursor. All schedule kinds use scheduled_jobs desired state and stable-key reconciliation. Run events deliver through encrypted webhook secrets and the shared address-pinned, redirect-revalidating, bounded HTTP client.

REST, MCP, schedules, durable runs/continuations, reattachment, and safe webhook delivery share one authenticated RLS control plane.

Key modules & tables

src/server/apikeys + user-keys — hashed grants, workspace resolution and live capability narrowing
src/app/api/v1 + src/server/openapi/spec.ts — 56 documented REST operations
src/app/api/mcp + src/server/mcp/server.ts — eight tools and two resources
src/server/agents/executor.ts + starts.ts + continuations.ts + runner.ts + reattach.ts — durable start/resume/stream lifecycle
src/server/jobs/index.ts — 15 queues and scheduled_jobs desired-state reconciliation
src/server/alerts/webhooks.ts + network/safe-fetch.ts — encrypted signing secret and safe bounded delivery

Deep dive: Ports →

See it end to end

These are the mechanics; the use-case guides walk a real workflow across the systems, and each demo workspace is a live example you can open and follow.