Module references

Intake

Read-only sync of external sources into the corpus. Intake is the module that turns a Confluence site, a SharePoint drive, a Slack workspace, or a batch of staged markdown into deduplicated, incrementally re-synced documents rows — each one carrying where it came from and what permissions it had there.

What it does

A connector source is a governed, workspace-scoped record: a type, non-secret config, an encrypted credential, and a resumable checkpoint. You register sources through the API (or your own UI), then trigger a sync; the crawl runs server-side in a worker, page by page, persisting its checkpoint after every page so a crash resumes rather than restarts.

Ingestion is idempotent by construction: documents are keyed on (source_id, external_id), and a document whose sourceVersion is unchanged is skipped outright — no re-chunk, no re-embed. Intake is strictly read-only against the remote system: the connector interface exposes only fullLoad, poll, listIds, and an optional getPermissions. Nothing writes back to the source.

The HTTP surface only enqueues. The worker holds a per-source PostgreSQL advisory lock for the whole operation, so full, incremental, prune, and permissions work on one source can never interleave — across queues or replicas.

Four source types

confluence, sharepoint, slack, and markdown_upload. The first three require a structured credential; markdown_upload is credential-less and reads files staged in its own config.documents array, flowing through the same normalize → chunk → persist path as a live connector.

Use it standalone

Adopting Intake alone means: an API key, one package import, and a worker process that drains the sync queue. Nothing else in the product has to be configured.

Give the key the right scopes
Reads (GET /api/v1/connectors, GET /api/v1/connectors/{id}) require scope data:read. Everything that changes state — create, update, delete, and triggering a sync — requires scope kb:write. There is no connector-specific scope; the connector-specific granularity lives in capabilities.
A workspace key (eli_sk_) is checked on scope alone. A delegated user key (eli_uk_) is additionally narrowed against the owner's live capabilities: connectors:read for the reads, connectors:write for create/update/delete, and connectors:execute for the sync trigger. A missing capability is a 403 with error: "insufficient_capability".
Import exactly one entry point
React: @eli-ai/react/connectors. Headless: @eli-ai/client/connectors. Types only: @eli-ai/contracts/connectors. None of the three pulls in another module's client, contracts subpath, or components.
Run a worker
The sync endpoint returns 202 and stops there — it validates the source, checks it is not paused, and enqueues. If nothing is draining the queue, sources stay at lastSyncedAt: null forever and no sync_runs row is ever written. This is the one out-of-process dependency Intake genuinely has.

the whole module, headlessts

import { createEliTransport } from "@eli-ai/client/core";
import { createConnectorsClient } from "@eli-ai/client/connectors";

const connectors = createConnectorsClient(
  createEliTransport({
    baseUrl: "https://eli.ai",
    apiKey: () => process.env.ELI_API_KEY!,
  }),
);

// kb:write — register a source. This does NOT start a crawl.
const source = await connectors.create({
  type: "confluence",
  config: { baseUrl: "https://acme.atlassian.net/wiki", spaceKeys: ["ENG"] },
  credential: { email: "svc@acme.com", apiToken: process.env.CONFLUENCE_TOKEN! },
});

// kb:write — enqueue the crawl. Resolves as soon as the job is queued.
await connectors.sync(source.id, { kind: "full" });

// data:read — poll the source and its sync-run history.
const detail = await connectors.get(source.id);
console.log(detail.status, detail.lastSyncedAt, detail.syncRuns[0]?.stats);

What Intake does NOT require

No ontology or entity extraction, no model provider or embedding key, no agent definitions, no eval config, no reranker, and no query surface. A source syncs, documents and chunks land, and the credential is encrypted at rest — all of that works with every other module absent. Two capabilities are genuinely deferred to queued follow-on jobs rather than done inline: entity extraction and embeddings — see How it composes below.

React components

@eli-ai/react/connectors exports exactly one component, EliConnectors, plus the types EliConnectorsProps, EliConnectorsLabels, and a re-export of ConnectorSyncKind. It is a client component ("use client") and reads its transport from EliProvider.

EliConnectors

Renders a titled surface containing a card grid — one card per connector source. Each card shows the source name (config.name when set, otherwise the type with underscores replaced by spaces), a status badge, a Last synced line, a credential badge, and the source's ACL-visibility badge. When canSync is on, each card also gets a kind picker (incremental / full) and a Sync button; the button is disabled while a sync is being queued, when the source is paused, and when the source has no stored credential. On success it refetches the list and calls onSyncQueued. It also renders its own loading, error, and empty states.

EliConnectors props

`labels`	Partial<EliConnectorsLabels>	Per-string overrides merged over the defaults. Every visible string except the two state labels comes from here.
`canSync`	boolean	Renders the per-card sync kind picker and Sync button. Defaults to true. Set false for a read-only view.
`onConfigure`	(connector?: ConnectorSource) => void	Supplies your own create/edit UI. Passing it adds a header button (called with no argument) and a per-card button (called with that source). Omitting it renders no configure affordance at all.
`onConnectorSelect`	(connector: ConnectorSource) => void	Fired when the card's name button is activated. Use it to open a detail view.
`onSyncQueued`	(result: SyncEnqueued) => void	Fired after the sync mutation resolves, with the 202 body. The list is refetched first.
`title`	string	Overrides the surface heading. Falls back to labels.title.
`description`	string	Overrides the sub-heading. Falls back to labels.description.
`loadingLabel`	string	Text for the loading state. Defaults to "Loading content connectors…".
`errorLabel`	string	Text for the list-level error state (which also renders a retry button). Defaults to "Content connectors could not be loaded.".
`onError`	(error: unknown) => void	Called for both the list query and the sync mutation.
`className`	string	Appended after the built-in "eli-root" class on the root section.

No emptyLabel prop

EliConnectors takes Omit<CommonSurfaceProps, "emptyLabel"> — the empty-state text is labels.empty, not a top-level prop. Passing emptyLabel is a type error.

EliConnectorsLabels (all keys required on the full type; pass any subset)

`title`	string	Default: "Content connectors".
`description`	string	Default: "Monitor governed content intake and start safe, server-side synchronizations.".
`configure`	string	Default: "Configure connector".
`sync`	string	Default: "Sync". Shown on the idle sync button.
`fullSync`	string	Default: "Full sync". The kind-picker option.
`incrementalSync`	string	Default: "Incremental sync". The kind-picker option, and the default selection.
`empty`	string	Default: "No content connectors are configured.".
`credentialReady`	string	Default: "Credential stored".
`credentialMissing`	string	Default: "Credential required".
`lastSynced`	string	Default: "Last synced". Prefixes the formatted lastSyncedAt.

provider wiring + EliConnectorstsx

import { createEliTransport } from "@eli-ai/client/core";
import { EliProvider } from "@eli-ai/react/provider";
import { EliConnectors } from "@eli-ai/react/connectors";
import type { ConnectorSource } from "@eli-ai/contracts/connectors";
import "@eli-ai/react/styles.css";

// The transport is the trust boundary: point it at a same-origin route that
// injects the Bearer key server-side. Never ship an eli_sk_ key to a browser.
const transport = createEliTransport({ fetch: sameOriginProxyFetch });

export function IntakePanel({ workspaceId }: { workspaceId: string }) {
  return (
    <EliProvider transport={transport} workspaceId={workspaceId}>
      <EliConnectors
        labels={{ title: "Sources", sync: "Run now" }}
        onConfigure={(connector?: ConnectorSource) => openSourceDialog(connector)}
        onSyncQueued={(result) => toast(`Queued a ${result.kind} sync`)}
        onError={(error) => reportError(error)}
      />
    </EliProvider>
  );
}

What this component deliberately does not do

It never creates, edits, or deletes a source, and it never renders sync-run history. It issues exactly two calls: connectors.list() and connectors.sync(id, { kind }). Create/update/delete is your UI driven through onConfigure plus the HTTP API below; history comes from GET /api/v1/connectors/{id}. That is a deliberate boundary — credential entry is left to the host application.

Headless, with your own markup

The same client the component uses is importable directly. Every method takes an optional EliCallOptions (workspaceId, signal, headers).

ConnectorsClient surfacets

import { createConnectorsClient } from "@eli-ai/client/connectors";

const client = createConnectorsClient(transport);

client.list();                            // → ConnectorSourceList
client.get(id);                           // → ConnectorSourceDetail (source + syncRuns)
client.create(input);                     // → ConnectorSource
client.update(id, input);                 // → ConnectorSource
client.remove(id);                        // → DeleteConnectorResult ({ deleted, id })
client.sync(id, { kind: "incremental" }); // → SyncEnqueued

HTTP API

Six operations under /api/v1/connectors. Every one is workspace-scoped: the key resolves a workspace (optionally selected with the X-Workspace-Id header) and all reads and writes run beneath row-level security, so a source in another workspace is simply a 404. Errors use the shared { error, message } envelope, with an issues array added on body-validation failures.

GET/api/v1/connectorsscope data:read · capability connectors:read

Lists the workspace's connector sources, oldest first, credential-masked. No connection to the remote system is opened.

Example requestbash

curl https://eli.ai/api/v1/connectors \
  -H "Authorization: Bearer $ELI_API_KEY"

Responsejson

{
  "items": [
    {
      "id": "01JZQ3W8N4B7K2XR9V0C5T6HMD",
      "workspaceId": "01JH8Z2P0KQ4RN6TS3WVYA1BCE",
      "type": "confluence",
      "config": { "baseUrl": "https://acme.atlassian.net/wiki", "spaceKeys": ["ENG"] },
      "syncState": {},
      "status": "active",
      "aclVisibility": "workspace",
      "hasCredential": true,
      "lastSyncedAt": "2026-07-30T11:04:02.118Z",
      "createdBy": "01JH8Z2P0KQ4RN6TS3WVYA1BCE",
      "createdAt": "2026-07-01T09:00:00.000Z",
      "updatedAt": "2026-07-30T11:04:02.118Z"
    }
  ],
  "total": 1
}

POST/api/v1/connectorsscope kb:write · capability connectors:write

Registers a source and returns 201 with the credential-masked row. Creating a source does not start a crawl. The credential is JSON-serialized and encrypted at write; it is never returned by any read.

Request body

`type`required	"confluence" \| "sharepoint" \| "slack" \| "markdown_upload"	Discriminates the body. Immutable afterwards — PATCH cannot change it.
`credential`	object	Structured per type: confluence { apiToken, email? }, sharepoint { tenantId, clientId, clientSecret }, slack { botToken }. Required for those three; markdown_upload accepts no credential key at all.
`config`	object	Non-secret source config. Validated per type: confluence requires a valid http(s) baseUrl (optional spaceKeys array); sharepoint requires a non-empty siteIds string array; slack and markdown_upload require nothing. Defaults to {}.
`status`	"active" \| "paused" \| "error"	Defaults to "active". A paused source rejects sync with 409.
`aclVisibility`	"workspace" \| "restricted"	Defaults to "workspace". "restricted" forces every document this source ingests to be restricted.

Example requestbash

curl -L -X POST https://eli.ai/api/v1/connectors \
  -H "Authorization: Bearer $ELI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "confluence",
    "config": { "baseUrl": "https://acme.atlassian.net/wiki", "spaceKeys": ["ENG"] },
    "credential": { "email": "svc@acme.com", "apiToken": "<atlassian-api-token>" },
    "aclVisibility": "workspace"
  }'

Responsejson

{
  "id": "01JZQ3W8N4B7K2XR9V0C5T6HMD",
  "workspaceId": "01JH8Z2P0KQ4RN6TS3WVYA1BCE",
  "type": "confluence",
  "config": { "baseUrl": "https://acme.atlassian.net/wiki", "spaceKeys": ["ENG"] },
  "syncState": {},
  "status": "active",
  "aclVisibility": "workspace",
  "hasCredential": true,
  "lastSyncedAt": null,
  "createdBy": "01JH8Z2P0KQ4RN6TS3WVYA1BCE",
  "createdAt": "2026-07-01T09:00:00.000Z",
  "updatedAt": "2026-07-01T09:00:00.000Z"
}

GET/api/v1/connectors/{id}scope data:read · capability connectors:read

One source plus its recent sync-run history, newest first (up to 50 runs). 404 when the id is unknown in this workspace.

Example requestbash

curl https://eli.ai/api/v1/connectors/01JZQ3W8N4B7K2XR9V0C5T6HMD \
  -H "Authorization: Bearer $ELI_API_KEY"

Responsejson

{
  "id": "01JZQ3W8N4B7K2XR9V0C5T6HMD",
  "workspaceId": "01JH8Z2P0KQ4RN6TS3WVYA1BCE",
  "type": "confluence",
  "config": { "baseUrl": "https://acme.atlassian.net/wiki", "spaceKeys": ["ENG"] },
  "syncState": {},
  "status": "active",
  "aclVisibility": "workspace",
  "hasCredential": true,
  "lastSyncedAt": "2026-07-30T11:04:02.118Z",
  "createdBy": "01JH8Z2P0KQ4RN6TS3WVYA1BCE",
  "createdAt": "2026-07-01T09:00:00.000Z",
  "updatedAt": "2026-07-30T11:04:02.118Z",
  "syncRuns": [
    {
      "id": "01JZR0M6C9F3H8QT4Y7XZB2NAV",
      "workspaceId": "01JH8Z2P0KQ4RN6TS3WVYA1BCE",
      "sourceId": "01JZQ3W8N4B7K2XR9V0C5T6HMD",
      "kind": "incremental",
      "startedAt": "2026-07-30T11:03:51.002Z",
      "finishedAt": "2026-07-30T11:04:02.118Z",
      "stats": {
        "docsSeen": 128,
        "docsAdded": 4,
        "docsUpdated": 9,
        "docsTombstoned": 0,
        "pages": 3
      },
      "error": null
    }
  ]
}

PATCH/api/v1/connectors/{id}scope kb:write · capability connectors:write

Edits a source. At least one field is required (an empty body is a 400). A present credential is re-validated against the source's stored type and re-encrypted; omitting it keeps the stored secret. Returns the masked row.

Request body

`config`	object	Replaces the stored config wholesale, then re-validated against the source's type.
`credential`	object	Any of the three structured shapes. Because type is immutable the body cannot discriminate, so a shape that does not match the stored type is rejected with 400.
`status`	"active" \| "paused" \| "error"	Pause or resume the source.
`aclVisibility`	"workspace" \| "restricted"	Change the default ACL policy.

Example requestbash

curl -L -X PATCH https://eli.ai/api/v1/connectors/01JZQ3W8N4B7K2XR9V0C5T6HMD \
  -H "Authorization: Bearer $ELI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "status": "paused" }'

Responsejson

{
  "id": "01JZQ3W8N4B7K2XR9V0C5T6HMD",
  "workspaceId": "01JH8Z2P0KQ4RN6TS3WVYA1BCE",
  "type": "confluence",
  "config": { "baseUrl": "https://acme.atlassian.net/wiki", "spaceKeys": ["ENG"] },
  "syncState": {},
  "status": "paused",
  "aclVisibility": "workspace",
  "hasCredential": true,
  "lastSyncedAt": "2026-07-30T11:04:02.118Z",
  "createdBy": "01JH8Z2P0KQ4RN6TS3WVYA1BCE",
  "createdAt": "2026-07-01T09:00:00.000Z",
  "updatedAt": "2026-07-31T08:12:44.500Z"
}

DELETE/api/v1/connectors/{id}scope kb:write · capability connectors:write

Removes the source and, by FK cascade, its sync_runs. Documents already ingested from it are retained and stay readable — their source_id simply dangles. Run a prune first if you want tombstones.

Example requestbash

curl -L -X DELETE https://eli.ai/api/v1/connectors/01JZQ3W8N4B7K2XR9V0C5T6HMD \
  -H "Authorization: Bearer $ELI_API_KEY"

Responsejson

{ "deleted": true, "id": "01JZQ3W8N4B7K2XR9V0C5T6HMD" }

POST/api/v1/connectors/{id}/syncscope kb:write · capability connectors:execute

Enqueues a crawl and returns 202. The body is optional — an empty body is parsed as an empty object. Enqueue only: the response says nothing about crawl progress, so poll GET /api/v1/connectors/{id} and read its syncRuns.

Request body

kind "full" | "incremental" Defaults to "incremental" — a delta poll windowed from the source's lastSyncedAt to now. "full" is a complete re-crawl, resumed from the stored sync_state checkpoint.

Example requestbash

curl -L -X POST https://eli.ai/api/v1/connectors/01JZQ3W8N4B7K2XR9V0C5T6HMD/sync \
  -H "Authorization: Bearer $ELI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "kind": "full" }'

Responsejson

{
  "enqueued": true,
  "sourceId": "01JZQ3W8N4B7K2XR9V0C5T6HMD",
  "kind": "full"
}

Two API-shape details worth pinning down

A paused source returns 409, coded invalid-config. The sync route maps the underlying connector error straight through, so the envelope is { "error": "invalid-config", "message": "Source <id> is paused" } at HTTP 409 — not a conflict code. Branch on the status, not the string.

Only full and incremental are reachable over HTTP. The data model and the worker both support prune (tombstone documents that vanished at the source) and permissions (re-read remote ACLs), and sync_runs.kind can hold all four — but the sync route's schema accepts only the first two, and SyncEnqueued.kind is typed to match. If you see a prune or permissions run in the history, it was enqueued server-side, not through this endpoint.

Data elements

Intake owns two tables and writes a defined set of columns on a third. All three are workspace-scoped under row-level security — every read and write goes through a workspace-bound transaction, so a cross-workspace id is invisible rather than forbidden.

Table	One row is	Intake's relationship	RLS
connector_sources	One configured external source: type, config, encrypted credential, checkpoint, status.	Owns — created and updated only by this module.	Workspace-scoped
sync_runs	One append-only sync attempt, with per-run stats and a failure message.	Owns — written by the worker, read back through the detail endpoint. Cascades on source delete.	Workspace-scoped
documents	One knowledge document. Shared with the rest of the product.	Writes the provenance and ACL columns below, plus the ordinary content columns, on ingest.	Workspace-scoped
chunks	One retrievable slice of a document.	Written inline during ingest via the shared chunker — same path a manual save uses.	Workspace-scoped
audit_log	One recorded action.	Appends `connector.create` / `connector.update` / `connector.delete` / `connector.ingest`. Credential values are never written — an update records only the names of the changed fields.	Workspace-scoped

connector_sources — the columns a consumer reads
Column	Type	What it means to a consumer
id	text (ULID)	The id used in every /connectors/{id} path.
type	enum	confluence \| sharepoint \| slack \| markdown_upload. Set at create, immutable after.
config	jsonb	Non-secret settings, shape per type (baseUrl/spaceKeys, siteIds/driveIds, channelIds/teamId/oldest, documents). Returned verbatim.
credential_enc	text · never returned	AES-256-GCM ciphertext. Stripped from every API read and replaced with the boolean hasCredential. Decrypted only in the worker at crawl time.
sync_state	jsonb	The resumable checkpoint (delta link / CQL watermark / per-channel cursors), rewritten after every crawled page. Opaque — treat it as a black box.
status	enum	active \| paused \| error. A failed sync sets error; a successful one sets it back to active. paused blocks the sync endpoint (409).
acl_visibility	enum	workspace \| restricted. "restricted" forces every document this source ingests to be restricted, and a content update can never downgrade it.
last_synced_at	timestamptz \| null	Set on a successful sync. Also the lower bound of the next incremental poll window; null means poll from the epoch.
created_by	text \| null	The actor id, or the key id when there is no actor.

sync_runs — append-only history
Column	Type	What it means to a consumer
source_id	text → connector_sources.id	FK, cascade delete.
kind	enum	full \| incremental \| prune \| permissions. Only the first two can be enqueued over HTTP.
started_at / finished_at	timestamptz	finished_at is null while the run is in flight — that is how you detect an active sync.
stats	jsonb	{ docsSeen, docsAdded, docsUpdated, docsTombstoned, pages }. Written once when the run finishes, on success and on failure alike.
error	text \| null	The failure message when the run threw; null on success. The job is retried by the queue, producing another row.

documents — the columns Intake sets
Column	Type	What it means to a consumer
source_id	text \| null	Which connector source produced this document. Null means a manual or vault document — that is the pre-Intake default and stays untouched.
external_id	text \| null	The document's id at the source. (workspace_id, source_id, external_id) is uniquely indexed where source_id is not null, which is what makes re-sync an upsert rather than a duplicate.
source_version	text \| null	The source's own version token (Confluence version number, Graph eTag, Slack ts), falling back to the content hash. Unchanged and live ⇒ the document is skipped entirely, so no re-chunk and no re-embed.
acl_visibility	text	workspace \| restricted, defaulting to workspace. Set from the source policy and any permissions refresh; a later content sync never downgrades a restricted document.
acl_principals	jsonb (string[])	Normalized principal ids carried over from the source. Retrieval filters on this pair in SQL before ranking.
path	text	Deterministic vault path — connectors/<source id>/<sanitized external id>-<8 hex>.md — so the markdown stays readable through the ordinary document surfaces.
content_hash / version	text / integer	Hash of the stored markdown; version increments on each changed re-sync.
deleted_at	timestamptz \| null	Set by a prune when the document vanished at the source. A later sync that sees it again clears it, reactivating the same row rather than creating a new one.

Prune has a deliberate safety property

A prune builds the complete set of currently-present external ids before deleting anything, so a source outage mid-enumeration aborts the run instead of mass-tombstoning the corpus. Tombstoning more than half of a source's live documents is logged as a warning but still executed.

How it composes

Intake is the first slice in the capability map, so most of what it gains from the others is downstream of the documents it produces. Two of those links are real dependencies for a specific feature; the rest are additive.

Atlas — entities and relations. Every ingested or changed document enqueues an extraction job. That job is what turns connector text into graph entities, and the enqueue is explicitly non-fatal: if nothing consumes it, ingestion still succeeds and the documents and chunks are all still there. Genuinely needs Atlas: anything that reads the entity graph over connector content.
Lens — retrieval and grounded answers. Chunks are persisted inline during ingest, so lexical retrieval over connector documents works as soon as a sync completes. Embeddings are a separate queued job, so the vector half of hybrid retrieval only covers connector documents once that job has run against a configured embedding provider. Retrieval is also where the ACL columns are enforced: the content-ACL predicate filters on acl_visibility and acl_principals in SQL before ranking, so a restricted source stays restricted end to end.
Lineage — provenance. Intake records a document revision on every connector upsert, tagged with the connector activity and a sourceId:externalId note. The lineage read joins connector_sources to classify a document's origin as connector, upload, or vault. Without Lineage you still have the raw columns; with it you get the assembled trail.
Warrant — governance. Connector documents are ordinary documents rows, so document verification and review SLAs apply to them exactly as they do to hand-authored ones. Nothing in Intake requires it.
Crucible and Ports. Purely additive: evals measure quality over whatever corpus exists, and agents and MCP read it. Neither is consulted during a sync.

Intake connectors are not Conduit connectors

/api/v1/connectors (this module) and /api/v1/data/connectors (Conduit) are different surfaces backed by different tables — connector_sources versus data_connectors. This module ingests content into the corpus; Conduit queries live systems at answer time and stores nothing in documents. They share the data:read scope on their list endpoints and the same encryption envelope for credentials, and nothing else. Adopting Intake does not adopt Conduit.

Deeper reading: Content connectors for per-source operational detail (rate limits, credential scopes), Connect Confluence, SharePoint & Slack for a walkthrough, and the OpenAPI reference for the wire contract.

Extending Intake

A plugin can add a source type — intakeConnectors opens the formerly closed Record<ConnectorType, factory>, so a contributed type resolves through the same buildConnector path, checkpoints, advisory lock and sync_runs rows as the four built-ins — or rewrite a document before it is persisted with intakeDocumentTransforms, which runs after normalisation and before hashing. Intake also emits document.ingested, document.updated, document.deleted, connector.source.* and connector.sync.completed / failed onto the plugin event bus. See the extension model for the contract and the trust boundary, and Build a plugin for a worked example.

Intake

What it does

Use it standalone

Give the key the right scopes

Import exactly one entry point

Run a worker

React components

EliConnectors

Headless, with your own markup

HTTP API

Data elements

How it composes