Content connectors

A knowledge base is only as good as what is in it, and most of what a team knows already lives somewhere else — Confluence spaces, SharePoint sites, Slack channels. Content connectors sync those sources into the same documents, chunks, and entity graph everything else runs on: read-only, incremental, resumable, and workspace-scoped under FORCE-RLS. These are distinct from data connectors, which run governed live SQL against operational databases — content connectors ingest documents, not rows.

One contract, four sources

Every source implements the same connector interface — the taxonomy is borrowed from Onyx (the MIT-licensed open-source connector suite), so governance, provenance, and citations are identical no matter where a document came from:

interface Connector {
  // Backfill everything, resumable from a checkpoint.
  fullLoad(checkpoint: Checkpoint | null): AsyncGenerator<DocBatch, Checkpoint>;
  // Incremental: only what changed in [start, end).
  poll(start: Date, end: Date): AsyncGenerator<DocBatch>;
  // Prune: the authoritative id set, to detect deletions.
  listIds(): AsyncGenerator<string[]>;
  // Per-document principals, normalized by the source permission adapter.
  getPermissions?(ids: string[]): Promise<DocPermissions[]>;
}

// Every batch yields normalized documents:
type Document = {
  sourceId: string; externalId: string; url: string | null; title: string;
  markdown: string; mimeType: string; sourceVersion: string | null;
  contentHash: string;
  aclVisibility?: "workspace" | "restricted";
  aclPrincipals?: string[]; updatedAt: Date;
};

Resumable, incremental sync

The sync_state jsonb on each source is the resumability contract, modeled on Airbyte's STATE message: after every document in a yielded page is durably written, the runner checkpoints that page's position — a SharePoint deltaLink, a Confluence CQL watermark, per-channel Slack cursors, or an opaque checkpoint blob. If one document fails, the checkpoint does not advance; a retry replays any earlier successful upserts idempotently instead of skipping the failed record. Every run appends a sync_runs row — an append-only ledger of kind (full · incremental · prune · permissions), timing, and stats (docs seen / added / updated / tombstoned).

Ingestion: upsert, dedup, tombstone

ingestConnectorDocument is the single write path, and it reuses the exact same primitives a hand-authored document uses — saveDocument, chunk persistence, entity extraction, and embedding — so a synced document is indistinguishable from one you typed:

• Upsert by identity — documents carry a source_id and external_id, and a partial unique index on (workspace, source_id, external_id) (where source_id is not null) makes re-sync a dedup: the second sighting of a page updates the same document rather than duplicating it. Manual and vault documents (null source_id) are untouched by this constraint.
• Content-hash change detection — each document records a source_version and content hash. If the hash is unchanged since the last sync, ingestion skips re-chunking, re-extraction, and re-embedding — the expensive work — so polling a large space every few minutes costs almost nothing when little changed.
• Soft-tombstone on delete — listIds gives the prune job the authoritative id set; anything present locally but absent upstream is soft-tombstoned, removing it from retrieval without hard-deleting the record. The run increments docsTombstoned only after a delete succeeds, so partial failures remain visible instead of inflating the completion count. A page that reappears upstream un-tombstones cleanly.
• Revision and audit continuity — connector create/update/reactivation and prune deletion append the same immutable document_revisions summaries and audit events used by manual edits. A reactivated row keeps its document identity and advances its version, preserving the end-to-end history instead of starting a second lineage.

A content change also notifies the semantic cache: ingestion calls the cache's cacheInvalidateOnDocChange hook, so a cached answer grounded in a page that a connector just re-synced is invalidated by the same event that updated the document.

Normalization and binary extractors

Native text and Slack mrkdwn normalize locally. Mammoth (.docx), Turndown (HTML), unpdf (PDF), and officeparser (other Office formats) are installed dependencies: PDF, DOCX, PPTX, and XLSX content extracts to markdown on the same normalization path every connector shares, and POST /api/w/{id}/import/binary accepts direct binary uploads. The imports stay availability-gated — a trimmed deployment that removes a package degrades to a clear, named skip — and a corrupt or unsupported file is reported as extraction failed rather than silently decoded, with a hard 30s extraction timeout as the backstop.

Credentials, egress & access

A source stores its secret as credential_enc — AES-256-GCM under the enc:v1 envelope, the same crypto the data connectors use. The API accepts a secret on create/update, stores only the ciphertext, and every read returns a masked placeholder plus a hasCredential flag; decryption happens only inside the sync runner at call time.

Permission refreshes carry both acl_visibility and normalized acl_principals into each document:

• Confluencereads each page's read restrictions and records canonical email, account, group, and space principals. A page becomes restricted when the provider reports explicit user or group grants.
• SharePoint reads Graph drive-item permissions and records email/user/group identities, inherited site/drive scopes, and organization-link tenant scope. The app itself uses Sites.Selected and requires explicit configured site/drive ids; site grants are a separate administrator action.
• Retrieval enforces before ranking.SQL admits workspace-visible rows or restricted rows whose GIN-indexed principals intersect the authenticated actor's normalized principals. The actor's app user id and email are automatic. Provider user/group ids require a trusted identity/group mapping to add them; an unresolved restricted grant fails closed instead of becoming workspace-visible.
• Shared workspace API keys and named background/system jobs are explicit workspace-service authority and can read restricted workspace content. Delegated user keys resolve the owner's current actor principals and stay filtered.

Managing sources

Sources are created and monitored on the workspace Connectors page (/w/<workspace>/connectors) — add a source per type, watch sync status and the sync_runs history, trigger a manual sync, and pause or resume. Over the API:

• GET/POST /api/v1/connectors — list (masked creds, data:read) and create (kb:write).
• PATCH/DELETE /api/v1/connectors/{id} — update config/creds or remove a source (kb:write).
• POST /api/v1/connectors/{id}/sync — enqueue a full or incremental sync (kb:write).

Research grounding