Skip to documentation
eli.aidocs

Platform

Scheduling, drift & alerts

A knowledge base degrades silently: models drift, evals regress, and a scheduled job that quietly stops running is the most dangerous failure of all. eli.ai closes that loop with recurring scans on a self-hosted cron, scan-staleness and worker-heartbeat signals, an alerts notification center, and outbound webhooks that push alerts into Slack, Discord, or any endpoint — signed and SSRF-guarded.

The pipeline

schedule → scan → threshold → alert → webhook.

Recurring scans

Scans run on a standard 5-field cron via pg-boss in the worker. Every recurring job first commits to scheduled_jobs, the authoritative RLS-scoped desired-state record of its cron, configuration, active flag, and last outcome. The broker projection happens after that commit; a failed handoff remains visibly reconcile_pending and the worker repairs it at boot and every minute. Two scan kinds ship:

  • Drift scans — periodically re-measure the KB and its answers for regressions (coverage gaps widening, groundedness slipping, retrieval quality moving) and flag material change.
  • Eval canaries — run a small golden set on a cadence so a provider or model change that quietly degrades quality is caught on a schedule, not by a client. See Evals & drift.

A third schedule kind, report, shares the same substrate: it generates a persisted report snapshot and queues its PDF render on a cadence, so the weekly client deliverable is already rendered before anyone asks. See Reports & PDF deliverables.

Recurring agent runs use that same registry. Each schedule has one deterministic, pg-boss-safe key; scheduled payloads return that key to the worker so success, budget skip, and failure update the exact desired-state row. Upgrades automatically rewrite legacy colon-delimited keys before applying the broker schedule.

Each run counts against the same budgets and cost caps as any model usage, and every scan writes a result row that also refreshes the scan-freshness signal below.

scan cadences (cron)text
0 * * * *     # hourly eval canary
0 7 * * *     # daily 07:00 drift scan
0 8 * * 1     # weekly Monday coverage drift

Scan freshness and worker availability

The failure that hurts most is the scan that stops running — a stalled worker, a crashed cron, a misconfigured schedule — because nothing fires to tell you. Each scheduled scan records its last result, and a staleness scan can raise kb.stale when ingestion or evaluation freshness exceeds the configured interval.

Use the independent worker heartbeat as the dead-man signal

A worker cannot run its own staleness scan after it has stopped. The resident worker therefore writes a separate PostgreSQL heartbeat, and yarn health:worker fails when the named instance is stale. The production Compose file wires that command into container health. Route that health failure to external monitoring; web readiness reports the heartbeat but does not fail when only the worker is stale.

Alerts notification center

When a scan breaches a threshold it raises an alert. The notification center is the one place every alert lands — drift regressions, canary failures, staleness. Two properties keep it usable at scale:

  • Deduplication. Each alert carries a dedupeKey, so a condition that recurs every hour coalesces into one open alert instead of flooding the center. Delivery is triggered by each firing path, so a recurring condition can still produce multiple webhook attempts against that one open alert.
  • Resolution. When the underlying condition clears, the alert's resolvedAt is set and it moves out of the active list, so the center reflects what is wrong now, not a growing scroll of history.

Outbound webhooks

Alerts push outward. Register webhook endpoints per workspace (webhook_endpoints) and each firing alert is delivered with a full attempt record in webhook_deliveries for retry and audit. Three transports ship:

  • Slack and Discord — formatted messages for their incoming-webhook APIs.
  • Generic — a JSON POST to any URL, for your own pipelines and paging systems.

HMAC signing

Every generic delivery is signed: an HMAC of the raw body under the endpoint's secret travels in a signature header, so a receiver can verify the payload actually came from your workspace and wasn't tampered with. Verify against the raw bytes before parsing. New and updated secrets are stored only as an AES-256-GCM enc:v1 envelope; API/UI reads expose hasSecret and a non-reversible masked hint, never ciphertext or plaintext. Production boot idempotently upgrades legacy plaintext rows, and unreadable/tampered envelopes fail delivery before any network call.

delivery headers (generic)text
POST https://your-endpoint.example/hooks/eli
X-Eli-Event: alert.raised
X-Eli-Delivery: 01J…              # alert/event id, stable idempotency key
X-Eli-Signature: sha256=<hmac>    # HMAC of the raw body under the endpoint secret

SSRF-guarded egress

Webhook URLs are attacker-influenced destinations, so every attempt uses the shared safe-egress client. Each DNS/redirect hop is resolved once, every returned address must be public, and the connection is pinned to that validated set while preserving TLS hostname verification. Private, loopback, link-local, metadata, and mixed public/private answers are refused. Redirects are manual, capped, and revalidated; webhook cross-origin redirects are refused. Requests have a 10-second deadline, a 1 MiB request cap, and a 64 KiB response cap.

Related