Skip to documentation
eli.aidocs

Intake

Knowledge base & vault

Everything eli.ai knows starts as plain markdown in a per-workspace vault. The vault is the source of truth for content; the database is a derived index that makes it searchable, linkable, and graph-traversable. This page explains how a document travels from disk to a citable, retrievable chunk.

Files-first, not database-first

Each workspace owns a directory of .md files plus attachments under DATA_DIR/vaults/<workspace>/. The files are Obsidian-compatible and git-able — you can edit them in eli.ai, in your editor, or sync them with Dropbox, and the app reconciles. The database stores what has no natural file form (chunks, embeddings, entities, relations, runs, evals) and a metadata index over the files (path, title, content hash, version).

Web edits write through to disk atomically (temp-file + rename) and never corrupt your vault. In local mode a filesystem watcher picks up external edits; on a remote host the server owns all writes and you trigger a manual rescan instead. Deleting a file is a soft-delete with a grace window keyed on the document's frontmatter id, so a folder rename never destroys provenance.

Document identity

Every document gets a stable ULID. On first save eli.ai injects it into frontmatter by textual splice — it never parses and re-serializes your YAML, because round-tripping mangles dates, comments, and key order across thousands of files. The splice is gated by a one-time consent prompt; if you decline, identity falls back to the content hash.

northwind/interviews/cfo-2026-03.mdmarkdown
---
id: 01JQ8Z6K3M9WXP0V7T2R4C5N8B
title: CFO Interview — Q1 planning
client: Northwind Trading Co
type: interview
tags: [finance, planning]
---

# CFO interview

We spoke with [[Dana Whitfield]] about the [[FY26 Budget]] and how
[[Northwind Trading Co]] plans capital allocation across business units.

A handful of frontmatter keys are meaningful: title, tags, client, and type. A bu (business-unit) mapping feeds the coverage matrix. Everything else is preserved verbatim.

Wikilinks

[[Wikilink]] syntax connects documents and seeds the entity graph. When you save, eli.ai resolves each link against known entity names and aliases: a match produces a mention and a references edge; an unresolved target becomes a stub entity, flagged and excluded from client-facing views until it earns real content. Because your explicit links are the highest-precision signal available, they build a connected graph with zero LLM cost — the graph exists even fully offline. See Entity graph.

Chunking

Retrieval and extraction operate on chunks, not whole documents. One deterministic module (remark mdast, heading-aware) splits each document into heading-scoped sections:

  • Target ~1,000 tokens per chunk, with a 200-token minimum and a 1,600-token hard cap.
  • Zero overlap — overlap would duplicate mentions and double extraction cost. The heading path (e.g. CFO interview › Capital allocation) is prepended as context to prompts and embeddings but is not stored inside the chunk text.
  • Offsets are absolutecharacter offsets into the original file, so a chunk's content is always exactly original.slice(startOffset, endOffset).
  • Chunk identity is (doc_id, sha256(normalized_text)). Renaming a heading is cheap; identity only changes when the text itself changes.

The dual index: FTS + vector

Each chunk is indexed two ways so retrieval can fuse lexical precision with semantic recall:

  • Full-text search (FTS) — a Postgres tsvector per chunk, queried with websearch_to_tsquery('english', …) and ranked by ts_rank_cd. Built synchronously on save.
  • Vector search — chunk embeddings stored in chunk_embeddings under an embedding_spaces row (provider + model + dimension), with a pgvector index. Embedding is the async half of the save path, dispatched to the worker.

Switching embedding models creates a new space and re-embeds in the background, so search never goes dark mid-migration. If no embedding provider is configured, retrieval simply runs FTS-only.

The save-path invariant

Saving a document is fully synchronous and never awaits the network: parse → chunk → FTS → wikilink resolution, all under 50 ms, works completely offline. Embeddings and LLM extraction are queued as background enrichment with a visible per-document freshness status. A slow or absent provider can never block a save.

Deletion is synchronous

Soft-deleting a document synchronously hard-deletes its derived rows (chunks, FTS, vectors) so retrieval can never surface deleted content. A later hard purge cascades through every verbatim-text copy — version snapshots, mention surface text, relation-evidence quotes — and emits a dated destruction record. This is what lets a consultant delete an off-the-record interview and be sure chat will not cite it an hour later. See Export & offboarding.

Next