Intake

Intake is the module that turns a Confluence site, a SharePoint drive, a Slack workspace, or a batch of staged markdown into deduplicated, incrementally re-synced documents rows — each one carrying where it came from and what permissions it had there.

What it does

A connector source is a governed, workspace-scoped record: a type, non-secret config, an encrypted credential, and a resumable checkpoint. You register sources through the API (or your own UI), then trigger a sync; the crawl runs server-side in a worker, page by page, persisting its checkpoint after every page so a crash resumes rather than restarts.

Ingestion is idempotent by construction: documents are keyed on (source_id, external_id), and a document whose sourceVersion is unchanged is skipped outright — no re-chunk, no re-embed. Intake is strictly read-only against the remote system: the connector interface exposes only fullLoad, poll, listIds, and an optional getPermissions. Nothing writes back to the source.

Use it standalone

Adopting Intake alone means: an API key, one package import, and a worker process that drains the sync queue. Nothing else in the product has to be configured.

React components

@eli-ai/react/connectors exports exactly one component, EliConnectors, plus the types EliConnectorsProps, EliConnectorsLabels, and a re-export of ConnectorSyncKind. It is a client component ("use client") and reads its transport from EliProvider.

EliConnectors

Renders a titled surface containing a card grid — one card per connector source. Each card shows the source name (config.name when set, otherwise the type with underscores replaced by spaces), a status badge, a Last synced line, a credential badge, and the source's ACL-visibility badge. When canSync is on, each card also gets a kind picker (incremental / full) and a Sync button; the button is disabled while a sync is being queued, when the source is paused, and when the source has no stored credential. On success it refetches the list and calls onSyncQueued. It also renders its own loading, error, and empty states.

Headless, with your own markup

The same client the component uses is importable directly. Every method takes an optional EliCallOptions (workspaceId, signal, headers).

HTTP API

Six operations under /api/v1/connectors. Every one is workspace-scoped: the key resolves a workspace (optionally selected with the X-Workspace-Id header) and all reads and writes run beneath row-level security, so a source in another workspace is simply a 404. Errors use the shared { error, message } envelope, with an issues array added on body-validation failures.

Data elements

Intake owns two tables and writes a defined set of columns on a third. All three are workspace-scoped under row-level security — every read and write goes through a workspace-bound transaction, so a cross-workspace id is invisible rather than forbidden.

Feature map

Behind the six operations and the single component, the runtime divides into five feature areas. Each area names the concrete modules and tables that deliver it, and the same names reappear in the code map and the internals section below, so every behavioural claim on this page can be traced to the code that enforces it.

System architecture

The topology below names the real components: route directories, exported server functions, the job queue, and the tables each stage writes. Rectangles are API routes and services, cylinders are tables and queues, and stadium nodes are the external systems a crawl reads. Intake calls no AI model anywhere in this path — embeddings and entity extraction are downstream jobs owned by other modules.

Where the logic lives

The code map below is the module boundary in file terms. Every path is real; table names live in the notes because tables are defined in the schema module, not in files of their own.

Primary runtime flow

The highest-value flow is a triggered sync: one HTTP call, then everything else in the worker. Message labels use the operation names from the HTTP API section above.

Internals and invariants

Source registry and credential custody

A source is created and edited through src/server/connectors/crud.ts, which validates non-secret config per type (a bad Confluence base URL or an empty SharePoint site list is a 400 at write time, not a failed crawl later), validates the credential shape against the stored type, and serializes the credential into an encrypted column. Reads go through a masking projection that replaces the ciphertext with a boolean.

• Invariant — Credential ciphertext never leaves the CRUD module unmasked: every exported read strips it to hasCredential, and decryption happens only in loadSourceForSync at crawl time, enforced in src/server/connectors/crud.ts.
• Invariant — Audit rows for source updates record only the names of the changed fields, never credential or config values, enforced in src/server/connectors/crud.ts.

Sync execution and checkpointing

The trigger validates the source and enqueues; the worker claims the job and hands it to a runner. Every runner first takes a session-level PostgreSQL advisory lock keyed on the source, then crawls with the adapter, persisting the generator's checkpoint after each page. Remote I/O happens outside any database transaction — only the short bookkeeping writes open one.

• Invariant — All four run kinds against one source serialize under a single advisory lock that spans queues and worker replicas, enforced by withConnectorSourceLock in src/server/connectors/sync.ts.
• Invariant — The resumable checkpoint is persisted after every crawled page, so a crash resumes rather than restarts, enforced by the crawl driver in src/server/connectors/sync.ts.

Normalization and deduplicated persistence

Adapters emit normalized documents; binary payloads flatten to markdown through availability-gated extractors. Persistence is a single upsert: frontmatter is injected, the document id is spliced in, content is hashed and chunked, and the row lands with its revision and audit entry in one workspace transaction. An existing live row with an unchanged source version is skipped before any of that work happens.

• Invariant — A live document whose sourceVersion matches is skipped outright — no rewrite, no re-chunk, no re-embed — enforced by persistConnectorDocument in src/server/connectors/sync.ts.
• Invariant — Chunk rows reconcile by identity (document, normalized content hash, position), so existing chunk embeddings survive exactly when they are still valid, enforced by persistChunks in src/server/ingest/chunking.ts.
• Invariant — A plugin document transform may rewrite or deliberately skip a document, but a transform that throws or returns garbage is ignored and the last good version persists; identity keys are pinned by the host, enforced in src/server/connectors/sync.ts.

ACL mirroring and prune lifecycle

Documents carry the visibility and principals their source reported. The permissions runner re-reads remote ACLs for every live document of a source; the prune runner enumerates the complete set of ids still present remotely and tombstones only what has verifiably vanished.

• Invariant — A content update never downgrades a restricted document, and a restricted source forces every document it ingests to restricted, enforced in the ACL merge of src/server/connectors/sync.ts.
• Invariant — A prune builds the complete present-id set before deleting anything; an enumeration failure aborts the run with zero tombstones, enforced by runConnectorPrune in src/server/connectors/sync.ts.

Run observability and enrichment handoff

Every attempt writes a run row at start and finalizes it with stats and an error string on completion — success and failure alike — while the source row tracks status and last-synced time. After a successful upsert, extraction and embedding jobs are enqueued for downstream modules and a cache invalidation is signalled for changed content.

• Invariant — Every sync attempt leaves an append-only sync_runs row with its stats, written by startSyncRun / finishSyncRun in src/server/connectors/sync.ts.
• Invariant — Extraction and embedding handoffs are non-fatal: an enqueue failure logs a warning and never fails ingestion, enforced in src/server/connectors/sync.ts.

For a self-hosting team the safe extension points are exactly the seams above: contribute a new source type through intakeConnectors (it resolves through the same buildConnector path in src/server/connectors/registry.ts and inherits the lock, checkpoints, and run rows), rewrite documents pre-persist with intakeDocumentTransforms, subscribe to the ingestion events on the plugin bus under src/server/plugins, or swap the vault body store in src/server/vault/store between its filesystem and Postgres implementations. The upsert key, the advisory lock, and the credential masking are load-bearing and not configurable.

How it composes

Intake is the first slice in the capability map, so most of what it gains from the others is downstream of the documents it produces. Two of those links are real dependencies for a specific feature; the rest are additive.

• Atlas — entities and relations. Every ingested or changed document enqueues an extraction job. That job is what turns connector text into graph entities, and the enqueue is explicitly non-fatal: if nothing consumes it, ingestion still succeeds and the documents and chunks are all still there. Genuinely needs Atlas: anything that reads the entity graph over connector content.
• Lens — retrieval and grounded answers. Chunks are persisted inline during ingest, so lexical retrieval over connector documents works as soon as a sync completes. Embeddings are a separate queued job, so the vector half of hybrid retrieval only covers connector documents once that job has run against a configured embedding provider. Retrieval is also where the ACL columns are enforced: the content-ACL predicate filters on acl_visibility and acl_principals in SQL before ranking, so a restricted source stays restricted end to end.
• Lineage — provenance. Intake records a document revision on every connector upsert, tagged with the connector activity and a sourceId:externalId note. The lineage read joins connector_sources to classify a document's origin as connector, upload, or vault. Without Lineage you still have the raw columns; with it you get the assembled trail.
• Warrant — governance. Connector documents are ordinary documents rows, so document verification and review SLAs apply to them exactly as they do to hand-authored ones. Nothing in Intake requires it.
• Crucible and Ports. Purely additive: evals measure quality over whatever corpus exists, and agents and MCP read it. Neither is consulted during a sync.

Deeper reading: Content connectors for per-source operational detail (rate limits, credential scopes), Connect Confluence, SharePoint & Slack for a walkthrough, and the OpenAPI reference for the wire contract.