C3 · Semantic layer & search

Notes are embedded as single 1536-dim vectors (openai/text-embedding-3-large via the Vercel AI Gateway, dimensions forced to 1536 in lib/ai/embeddings.ts) by three producers — a 30s-debounced auto-embed queue fired on every note save, a batch HTTP route driven by a settings-page indexer card, and a standalone backfill CLI — and stored per mode: a note_embeddings SQLite side table of Float32 BLOBs locally, the notes.embedding pgvector column in SaaS. At query time an FTS pass (SQLite FTS5/bm25 locally, Postgres tsvector RPC in SaaS) and a vector KNN pass (brute-force cosine locally, notes_knn pgvector RPC with HNSW in SaaS) are fused with weighted Reciprocal Rank Fusion (k=60, weights fts 0.7 / vector 0.3) and exposed via POST /api/search/hybrid plus debug explorer UIs at /search (local) and /w/[workspace]/search (SaaS).

Modes: local (FTS5 + BLOB brute-force cosine) and saas (tsvector + pgvector HNSW), same orchestrator · Key dirs: lib/search/, lib/ai/ (embeddings, auto-embed-queue, chunking), lib/storage/local/ + lib/storage/supabase/ (search, embeddings, knn), app/api/search/hybrid/, app/api/ai/embed/, components/search/, components/ai/, scripts/ · Depends on: C1 storage adapters + schema, C4 AI gateway/key resolution, C5 MCP rate limiter, C7 auth/RLS in SaaS · Feeds: C4 RAG chat/research retrieval, C5 agent search tools, C2 search UX entry points

Capabilities

The RRF formula (as implemented in lib/search/rrf.ts:86-119): for each document d, score(d) = Σ over lists L of w_L × 1 / (k + rank_L(d)) where rank_L(d) is d's 1-based rank in list L. Constants: RRF_DEFAULT_K = 60 (Cormack et al. 2009, rrf.ts:32) and HYBRID_DEFAULT_WEIGHTS = { fts: 0.7, vector: 0.3 }(spec §7.3 "OpenClaw" BM25-70/vector-30 split, rrf.ts:35). RRF is score-scale invariant (only ranks matter), so bm25 scores and cosine similarities never need normalising; a doc appearing in only one list still scores from that list. Ties are broken by id ascending for determinism (rrf.ts:114-118); k <= 0 throws; rank <= 0 entries are skipped defensively (rrf.ts:102).

System view — simplified

This cluster is eli's retrieval brain: an indexing side that turns every saved note into exactly one 1536-dim embedding, and a query side that runs lexical and semantic retrieval in parallel and fuses the two rankings with weighted RRF. Standalone, it is already a complete search product for a markdown corpus: point the backfill CLI at a vault, and POST /api/search/hybrid plus the /search explorer give ranked, snippet-annotated, explainable results with per-source rank attribution — no chat, no agents, no SaaS required. It degrades gracefully along two axes: with no AI key or no embeddings, every query silently falls back to FTS-only; with kind='fts' it costs zero gateway calls. The same orchestrator (lib/search/hybrid.ts) serves both storage modes by branching on the adapter class, so callers never see the local/SaaS divergence.

System view — detailed

Data flow — simplified

The load-bearing runtime flow is a hybrid query: the route validates and rate-limits, then hybridSearch runs the lexical pass first (fetching limit*2 candidates so fusion has depth), embeds the query text through the same embedTexts helper used for indexing, and runs KNN twice — restricted to the FTS candidate ids for precision, then globally when results run short, so purely-thematic matches still surface. Fusion is pure rank math (no score normalisation), and vector-only hits get their title/path hydrated per-id before the wire response.

Data flow — detailed

(a) Note save → auto-embed → vector stored — the indexing side, with local/SaaS persistence divergence:

(b) Hybrid query → FTS candidates + KNN → RRF fuse → ranked hits — the retrieval side, with local/SaaS branches at auth, FTS, and KNN:

Works separately / works together

Standalone

With every other cluster turned off, this cluster is a self-contained search engine over a markdown corpus. The backfill CLI (pnpm embed:backfill --vault <path>) indexes a vault with zero HTTP surface; the FTS5 index is maintained autonomously by SQL triggers whenever the notes table changes; POST /api/search/hybrid serves ranked results to any HTTP client; and the /search explorer makes ranking behaviour inspectable (per-source ranks, contributions to 5 decimals, FTS-only vs vector-only tinting). Even without an AI key the cluster still works: kind='fts' never touches the gateway, and hybrid queries silently degrade to FTS-only (lib/search/hybrid.ts:143-148). lib/search/rrf.ts is a dependency-free pure library usable in isolation. pnpm bench-search provides an offline FTS latency gate.

Composed

• C2 Authoring UX → this cluster: saveNoteAction is the sole enqueuer of the auto-embed queue (app/_actions/notes.ts:96-105 → scheduleEmbed); deleteNoteAction → cancelEmbed (notes.ts:236-237). createNoteAction does notenqueue — new notes get embedded only after their first save, an "Index pending" run, or the CLI. The command palette's quick search is plain FTS via the searchNotesAction server action (limit 20; empty query lists the 20 most-recent notes via listNotes, app/_actions/notes.ts:16-29) — it never touches the hybrid pipeline — plus a palette nav item that jumps to /search (components/command-palette.tsx:118,193). /api/search/hybrid is the only route under app/api/search/; there is no plain /api/search endpoint.
• C1 Core data & storage → this cluster: the storage adapters supply the FTS lists — LocalAdapter.search() (FTS5/bm25, lib/storage/local/search.ts) and SupabaseAdapter.search() (notes_search RPC); hybridSearch branches on adapter instanceof LocalAdapter / SupabaseAdapter. The FTS5 triggers that keep notes_ftsin sync ride on C1's note writes and the chokidar reindex path.
• C4 AI platform → this cluster: lib/ai/resolver.ts, gateway-config.ts, gateway-opts.ts, and lib/ai/models.ts supply keys, persisted gateway options, and the pinned embed model to every embedTexts call.
• This cluster → C4 AI platform: RAG chat calls hybridSearch(lastUserTurn, topK=8) for retrieval (app/api/ai/chat/route.ts); the OpenAI-compatible /v1/chat/completions route does the same with eli_top_k default 8 (0 disables RAG) and inlines note bodies sliced to 4000 chars (app/v1/chat/completions/route.ts:174-190); buildVaultTools registers a search_notes AI-SDK tool (hybrid, default limit 10, max 50) for the chat tool loop (lib/ai/tools.ts:127-156). The chunking helper owned here is consumed by /api/ai/research (maxChunks 30) and /api/ai/chat[/stream] (cap 24 chunks) for context-window management — never for embeddings.
• C5 Agent surface ↔ this cluster: the MCP rate limiter (lib/mcp/ratelimit.ts, built for PAT limits) is reused verbatim by /api/search/hybrid and /api/ai/embed, so web search/embed traffic and MCP PAT traffic share one in-memory backend (different keys). The MCP search tool takes kind: fts|semantic|hybrid (default fts = plain storage.search); hybrid/semantic dynamic-import hybridSearch, forward authInfo.workspaceId in SaaS, and return structuredContent hits with fused_score/fts_rank/vector_rank/contributions (lib/mcp/tools/search.ts:54-103). memory_search uses plain storage.search with a cap×5 over-fetch then path-scope filter (lib/mcp/tools/memory.ts:86-87).
• C7 SaaS collaboration → this cluster: lib/auth/supabase.ts provides the user-scoped client (RLS gating for FTS reads and notes_knn) and the service-role client (embedding writes); /api/ai/embed additionally does an explicit memberships-table check (403).
• Settings UI → this cluster: the AI settings page consumes GET /api/ai/embed (local progress) and renders EmbedIndexerCard, which drives POST /api/ai/embed.

Load-bearing details

• One vector per note, not per chunk. Input is title + '\n\n' + body truncated to 30,000 chars in every producer (lib/ai/auto-embed-queue.ts:90, app/api/ai/embed/route.ts:128-130 and :204, scripts/embed-backfill.mjs:109 and :157). lib/ai/chunking.ts is never used for embeddings — content past 30k chars is invisible to semantic search.
• RRF constants pinned: k=60 (lib/search/rrf.ts:32, test-pinned at rrf.test.ts:188-190), weights fts 0.7 / vector 0.3 (rrf.ts:35); score = Σ w × 1/(k + rank); id-ascending tie-break (rrf.ts:114-118).
• SaaS hybrid short-circuit: with kind='hybrid', zero FTS hits, and a non-Local adapter, hybridSearch returns [] before running the vector side (lib/search/hybrid.ts:79-81) — SaaS hybrid can never surface vector-only matches on an FTS-empty query, while local can. The Semantic tab (kind='semantic') bypasses this.
• Two-phase KNN in both modes: candidate-restricted KNN first (precision), then a global top-up appended when results < limit; both calls use limit*2 (hybrid.ts:98-141).
• Local vector store deliberately avoids sqlite-vec:brute-force cosine over Float32 BLOBs, justified as "~30ms for ≤50k notes on M1" (lib/storage/local/embeddings.ts:16-21); bytes decoded via DataView with explicit little-endian for cross-CPU portability (embeddings.ts:45-53). SaaS uses pgvector cosine (<=>) with an HNSW index; the notes_knn RPC is SECURITY INVOKER so RLS applies, and returns similarity = 1 − distance to match the local contract (lib/db/migrations/pg/0013_notes_knn.sql:26-40).
• Local KNN asymmetry: the global-scan branch joins notes and excludes trashed rows (local/embeddings.ts:237-245) but the candidateNoteIds branch does not filter trashed_at (:227-235); rows with mismatched model or dim are silently skipped (:249-252).
• Auth split on /api/search/hybrid: local has NO auth (rate key literal 'local', route.ts:41-42); SaaS requires a session (401) but no explicit membership check — isolation relies on RLS through the user-scoped client used by both the FTS RPC and notes_knn (route.ts:43-49, knn.ts:9-11). /api/ai/embed is stricter in SaaS: session (401) AND memberships check (403) before rate limiting (embed route.ts:76-97); reads via RLS, writes via service role to avoid an embedding-column UPDATE RLS policy (supabase/embeddings.ts:6-14).
• Rate limiting: checkRateLimit, 60/min + 10,000/day in-memory globalThis-pinned buckets (lib/mcp/ratelimit.ts:37-38, 101-105); local mode uses the single shared key 'local' on both routes, so local search and embed calls drain one common budget. ELI_RATELIMIT_BACKEND=redisthrows "not shipped yet" (ratelimit.ts:186-192).
• Query embeddings never use workspace BYOK: SaaS workspace BYOK needs workspaceId AND provider AND ELI_KMS_KEY (resolver.ts:41), but hybridSearch passes only workspaceId (hybrid.ts:91-93) — query embedding falls to local BYOK or AI_GATEWAY_API_KEY. POST /api/ai/embed does accept a provider param and can use BYOK for indexing. embedTexts forces dimensions: 1536 via providerOptions.openai and throws if the returned vector is not 1536-dim (lib/ai/embeddings.ts:60-87).
• Auto-embed is best-effort: errors only console.warn (auto-embed-queue.ts:64-70), no retry, no persistence (a worker recycle loses at most one cycle, module docstring lines 28-31). Sub-50-char notes are never embedded (:91).
• Every batch write is an idempotent upsert: local ON CONFLICT(note_id) DO UPDATE in one transaction (local/embeddings.ts:126-141); SaaS per-row UPDATE (supabase-js has no multi-row distinct-payload update, supabase/embeddings.ts:34-46) which throws on the first row error.
• Backfill pagination reality: local caps at listNotesMissingEmbedding's default 5000 per invocation (embed-backfill.mjs:94, local/embeddings.ts:162) — >5000-note vaults need re-runs; SaaS does one unpaginated SELECT with no .limit/.range (embed-backfill.mjs:134-138), implicitly capped by PostgREST max-rows. Batch defaults differ: CLI 32 (embed-backfill-args.mjs:37) vs HTTP route 50 (embed route.ts:131, 205); both max 500. Missing-detection is also asymmetric: local is model-aware (missing = no note_embeddings row for the current embed model, so a model change re-embeds everything), while SaaS treats any falsy notes.embedding as missing regardless of model (embed-backfill.mjs:143) — a model migration never triggers SaaS re-embeds.
• FTS local side: notes_fts is a contentless FTS5 virtual table (content='notes', content_rowid='rowid', columns title/body_md/type, tokenize='unicode61 remove_diacritics 2') synced by AFTER INSERT/UPDATE/DELETE triggers on notes, with a 'rebuild' maintenance command — all pinned by lib/db/fts.test.ts:71-139; tags are deliberately not indexed inline — filtered via a note_tags EXISTS join to keep tag changes O(1) (lib/db/migrations/sqlite/0001_fts.sql:3-18, local/search.ts:72-76). Search orders by bm25(notes_fts)with sign-flipped score ("bm25 is negative-better; flip sign", lib/storage/local/search.ts:85-102) and <mark>-wrapped snippets (snippet(notes_fts, -1, '<mark>', '</mark>', '…', 12)); trashed rows excluded.
• FTS query building (local): ftsQuery strips every FTS5 metacharacter (" * ( ) : ^ ~ + - , → space), splits on whitespace, and turns each token into a quoted prefix match "token"* joined with AND (typeahead-friendly, local/search.ts:25-33) — so hyphens/apostrophes become token splits and user-supplied phrase queries are impossible. Nothing usable left → null → empty result set without touching SQLite; a fully-empty query instead returns the 20 most-recently-updated non-trashed notes; limit clamped 1..100, optional type=/tag= filters (local/search.ts:41-77).
• FTS SaaS side: notes.fts is a STORED generated tsvector — to_tsvector('english', coalesce(title,'') || ' ' || coalesce(body_md,'')) — with a GIN index, recomputed by Postgres on every row write (no trigger code, lib/db/migrations/pg/0000_initial.sql:75-78,103). The notes_search(p_workspace_id, p_query, p_limit, p_type, p_tag) SECURITY INVOKER RPC parses via websearch_to_tsquery('english', …), orders ts_rank_cd desc then updated_at desc, filters tags via frontmatter->'tags' ? p_tag, clamps limit 1..100 — it replaced an injectable literal-interpolation search path (pg/0010_notes_search.sql). The surfaced SaaS score is row-position-derived (1/(idx+1)), not raw ts_rank_cd, and the snippet is a plain 200-char body_md prefix — no <mark> highlighting, unlike local (lib/storage/supabase/queries.ts:359-387).
• Wire shape: {query, hits:[{noteId, title, path, snippet, fusedScore, contributions, ftsRank, vectorRank}]} (hybrid route.ts:65-77); the explorer renders contributions to 5 decimal places and tints rows blue (FTS-only) / amber (vector-only) (hybrid-explorer.tsx:154-219).
• Vector-side failure is invisible: hybridSearch swallows ALL vector-side errors with a bare catch and degrades to FTS-only (hybrid.ts:143-148); for kind='semantic' the same catch yields an empty result set rather than an error.
• Validation limits: hybrid q max 1000 chars, limit max 100 (route.ts:18-24); embed noteIds max 5000, all:true SELECT limit 5000. Both routes export dynamic = 'force-dynamic'.
• Bench budget: pnpm bench-search exits 3 when FTS p95 exceeds ELI_PERF_P95_BUDGET_MS (default 200 ms) so CI can fail on regressions (scripts/bench-search.mjs:88-93).

Gaps and partially wired

• Stale comments contradict shipped code: lib/search/hybrid.ts:21-25 and app/(saas)/(app)/w/[workspace]/search/page.tsx:10-16both claim SaaS "doesn't yet expose knnByEmbedding" / "falls back to FTS-only", but hybrid.ts:117-141 fully implements SaaS vector search via the notes_knn pgvector RPC (migration 0013). The MCP search tool's agent-facing description has the same problem: it tells models semantic/hybridare "not yet implemented — lights up in Phase 5" (lib/mcp/tools/search.ts:36-37,44) even though both kinds work — an agent trusting the description will avoid working functionality. Verified against source.
• StorageAdapter.semanticSearch is a dead interface method: declared in lib/storage/adapter.ts:125 but both adapters throw NotImplementedError('semanticSearch', 'Phase 5 (embeddings)') (local.ts:244-245, supabase.ts:133-134); the hybrid module never calls it, going straight to the knnByEmbedding/knnNotes helpers instead.
• versionId is a dead field: scheduleEmbed's ctx documents it as "used to skip stale embeds" (auto-embed-queue.ts:44) but runEmbed never reads it.
• No chunk-level retrieval:whole-note vectors with a 30k-char truncation mean long-note tails are unsearchable semantically; the chunking helper exists but is wired only into C4's chat/research map-reduce.
• createNoteAction never schedules an embed — brand-new notes are semantically invisible until first edited, indexed via the settings card, or backfilled.
• Auto-embed has no retry and no queue persistence — a failed gateway call or a recycled worker silently drops the embed until the next save.
• Redis rate-limit backend is a stub: ELI_RATELIMIT_BACKEND=redisthrows "not shipped yet" (lib/mcp/ratelimit.ts:186-192); everything is in-memory and per-process.
• Shared local rate bucket: all local search/embed/chat traffic drains the single 'local'key — a burst of hybrid queries can 429 the same solo user's embed calls.
• SaaS backfill is unpaginated: one service-role SELECT of all workspace notes with no .limit (embed-backfill.mjs:134-138) — large workspaces are implicitly truncated by PostgREST max-rows, with no warning. Local backfill needs manual re-runs past 5000 missing notes.
• Arg parser quirk: flag() returns true when a flag has no value or is followed by another --flag (embed-backfill-args.mjs:13-19), so --vault --batch 5 yields vault === true (truthy) and kind 'local' with a boolean vault path; --batch with no value silently falls back to 32.
• Trashed notes can surface in the local candidate-restricted KNN branch (no trashed_at filter, local/embeddings.ts:227-235), unlike the global branch and the SaaS RPC.
• SaaS hybrid-with-empty-FTS returns nothing (hybrid.ts:79-81) — an intentional asymmetry vs local, but it makes the Hybrid tab on SaaS blind to vector-only matches; users must switch to the Semantic tab.
• Bench CLI covers FTS only — there is no latency benchmark for the vector or fused pipeline (scripts/bench-search.mjs calls adapter.search exclusively).

See also: 01-core-data-storage (adapters, schema, FTS triggers), 02-authoring-ux (save path), 04-ai-platform (gateway, keys, RAG consumer), 05-agent-surface (rate limiter, agent search tools), 07-saas-collaboration (RLS, memberships), 10-end-to-end-journeys (full save→embed→search trace), and the platform overview.