Learned LLM router

Multi-provider AI answers which providers a workspace may use; it does not answer which model this request deserves. A frontier model on a one-line summarization burns money; a cheap model on a hard reasoning chain burns quality. The learned LLM router is an optional sidecar (services/llm-router, Python + FastAPI) that closes that gap: for each request it predicts the quality score every catalog model would earn, estimates each model's dollar cost, and picks the model that maximizes quality-per-dollar subject to a quality floor. Published routers (RouteLLM, RouterBench) support 35–85% cost reduction at near-quality-parity — but the router treats that as a claim to benchmark, not a promise.

How a request becomes a decision

Each candidate model is an arm. The scorer is one LightGBM regressor over [MiniLM embedding (384) ⊕ ~45 engineered features ⊕ model one-hot] trained to predict the judge score that model would earn on the query — so scoring a slate is one feature computation plus N cheap forward passes, sub-millisecond on CPU. The engineered features read the request directly: token counts, fenced code blocks, structured-output demands (JSON/table/schema), conversation depth, math symbols, language, question type, required capabilities. Before any training artifact exists, the scorer falls back to documented static tier priors, so the router works out of the box and reports itself as scorer: "heuristic" on /health.

The policy: quality-per-dollar under a floor

• Eligibility first.A model must cover the request's required capabilities (vision, tools, JSON mode) and fit any max_cost_usd budget. If that leaves nothing, the router returns the safe default rather than no answer.
• The floor always applies. Only models whose predicted quality clears 0.62 are considered. The floor is deliberately conservative, because over-aggressive cheap routing is the classic router failure mode. If nothing clears it, the router takes the highest-quality eligible model — it never silently settles.
• Then maximize quality-per-dollar among the models that cleared the floor. That is the default objective, and the reason a mid-tier model routinely beats both the frontier model (which costs far more for a marginal quality gain) and the cheapest one (which fails the floor).
• A per-request min_quality flips the objective. Set it and the router switches to cheapest model clearing that bar— the "I need at least this good, spend nothing extra" mode.
• Low-confidence escalation. Confidence is the margin between the top two predicted qualities; below 0.05 the router escalates to a safe mid-tier default rather than guessing between near-ties.
• Fallback chain. Every decision carries up to three next-best eligible models, so the execution gateway can cascade on API errors, timeouts, or refusals.

The catalog itself is declarative — config/models.yamllists each model's provider, litellm string, per-token prices, tier, and capabilities. Adding a model is a config edit plus a golden-set re-run; no code changes.

Trained offline, measured like a benchmark

There is no live traffic to learn from on day one, so the scorer bootstraps from a 2,000-example golden set stratified over ~15–18 task classes: every enabled model answers every query, a frontier judge (from a different model family, with verbosity and position bias mitigations, every judgment cached) scores every answer, and verifiable classes use free programmatic grading instead. The one-time bill is roughly $50–150, fully cached. The trained router is then judged as a curve: sweeping willingness-to-pay traces its cost-quality Pareto frontier against always-frontier, always-cheapest, random, and a keyword-heuristic baseline — with bootstrap confidence intervals, and acceptance gates including ≥30% cost reduction at quality parity with always-frontier and <50 ms p95 router overhead. This mirrors how evals gate the rest of the platform: no metric, no rollout.

Shadow-first integration

LLM_ROUTER_MODE=shadow                # off (default) | shadow | active (reserved)
LLM_ROUTER_URL=http://localhost:8100  # the sidecar's FastAPI service

In shadow mode the host asks the router what it would have picked — a fail-soft POST /route started before generation and capped at 800 ms, so it scores concurrently with the answer and can never delay or fail it. The would-have-been decision is logged next to the model that actually answered. Once the answer resolves, the host reports a RewardSignal to POST /feedback: the model used, the realized cost, and an implicit quality reward derived from the answer's groundedness and policy verdict. A week of that yields routing mix, predicted-vs-realized quality, and would-have-saved cost before any user-facing change — and it is exactly the Phase 2 substrate: a contextual bandit, warm-started from the supervised scorer, learns from rewards the shadow phase is already collecting.