Who decides which model serves a request, where cost is actually computed, and why routing is per-conversation — design notes from studying the space.
Every serious LLM product eventually grows the same organ: a routing layer. You have multiple models — different providers, different sizes, different prices — and every request has to land on one of them, under constraints of cost, latency, quality, and who the customer is. These are my working notes on how I'd design that system, built up from studying how the large AI platforms do it and pressure-testing the design against the failure modes.
The one-sentence architecture: the edge resolves who you are (facts), the router decides which model (policy), the model gateway serves and meters, and the billing pipeline rates — with a versioned config store as the shared backbone. Everything below unpacks why the responsibilities split exactly there.
Two client populations enter differently — retail through a chat service that owns conversation history, API customers straight into the gateway with their own history — but converge on the same spine. Scale assumption for everything below: roughly 10k RPS peak, streaming responses over SSE, with end-to-end latency dominated by the model itself, which is what buys the routing layer its 10–20ms decision budget.
The most important design line in the system is between the API gateway and the router, and it's the line between facts and policy.
The gateway is per-tenant and model-agnostic. It terminates retail JWTs and business API keys and normalizes both into a signed context: {tenant_id, tenant_type, tier, region, experiments_eligible}. It enforces dual rate limits — requests per minute and tokens per minute, because cost lives in tokens, not calls — plus spend caps fed back from billing, abuse pre-filtering, and the unglamorous work of holding 30-second SSE streams open and draining them gracefully on deploys. It can be built on off-the-shelf infrastructure (Envoy, Kong); the routing intelligence deliberately stays out of it.
The router is per-request and model-aware — and stateless. It reads the gateway's facts, looks up what they mean in the config store, and decides which model serves the request.
The test I use: does this change when a model launches? Then it's router config. Does it change when a user upgrades their plan? Then it's a gateway fact. The same human holding a retail subscription and an API key is two tenants, keyed by credential.
Order matters here — each step constrains the next:
auto continues to selection.hash(user_id + salt) into buckets. Experiments can override defaults; they can never override hard filters.{decision, reason, config_version, bucket} — the decision must be deterministic given (user, request, config version), or you can't debug it and can't trust your experiment data.The router is also a brand-new single point of failure that didn't exist before you added it. It runs as stateless replicas, and the gateway holds a static default route so that total router failure degrades to "everyone gets the safe default model" rather than an outage.
Everything slow-changing that expresses intent lives in one versioned store: the model registry (endpoints, context limits, capabilities, lifecycle state), the pricing catalog (versioned and effective-dated), tier-to-policy mappings, routing thresholds, fallback chains, breaker parameters, experiment definitions, and capacity topology. Writes are validated and dry-run; pushes propagate by watch/subscribe; every consumer caches last-known-good; rollback is one click to a previous version. Model launches, price changes, and routing changes all happen without a deploy.
Just as important is what's deliberately not in it: billing state, conversation history, live health signals, and secrets — each of which has a rightful owner elsewhere.
Below the router sits an adapter layer that speaks each provider's API (or adopts the OpenAI-compatible format as a lingua franca) behind one unified schema. It handles streaming passthrough with one iron rule — retry only before the first token, never mid-stream — plus fallback chains, per-provider circuit breakers, and token-bucket budgets per provider key.
Its most important output is the usage event: {request_id, requested_model, served_model, input/output/cached tokens, pricing_version}. This is the source of truth for what happened — facts only, no pricing math. Which leads to the part of the design I find most underrated.
| Stage | Where | Purpose | Authoritative? |
|---|---|---|---|
| Estimate | Router / gateway, pre-request | Admission control, cost-aware selection | Never billed |
| Measurement | Model gateway, post-response | The usage event — token facts | Facts, not cost |
| Rating | Billing aggregator, async | Dedupe by request_id, join versioned price catalog, produce invoices/credits | Yes — the only one |
| Spend counter | Real-time (Redis-style) | Fast approximate cap enforcement at the gateway | Reconciled in batch |
Keeping these four honest about their roles prevents an entire family of bugs where two components disagree about what something cost. Rating is replayable — at-least-once event delivery plus idempotent dedupe gives effectively-once billing, and a price-catalog correction can re-rate history.
Edge cases worth having answers for: a stream disconnect bills the tokens actually generated (flush the usage event on abnormal exit); a fallback charges the lower of requested vs. served model; auto mode bills a blended flat rate, so routing savings become margin; and joining provider-side costs per tenant yields gross margin per customer — which feeds back into routing policy.
Models are stateless, so history is resent every turn. Retail gets a chat service and conversation store (append-only messages(conversation_id, seq, role, content, tokens, model_id), partition-keyed, partial responses persisted on disconnect, token counts stored at write time); API customers own their own history.
Context assembly is a budgeting problem: budget = context limit − system prompt − reserved output, filled by a sliding window, then rolling summarization (a cheap model compressing old turns), then retrieval. Two couplings back to routing matter: context length is a hard filter (long conversations exclude small models), and prompt caching makes stickiness an economic requirement, not a preference. And since cost grows roughly quadratically with turns, summarization is a cost lever — not just a trick for fitting the window.
Randomize on user_id, not per-request — consistent UX and clean statistics, at the price of slower ramps; chat products take that trade. Overlapping experiment layers allow concurrent tests without interference. Assignments stay sticky within a conversation. Measure latency, cost per request, error rate, thumbs ratio, regeneration rate, and retention, backed by offline LLM-judge evals; wrap it in guardrail metrics with auto-shutoff, and roll out as shadow traffic → 1% canary → ramp. Shadow traffic in particular is how a new model gets evaluated on real requests with zero user exposure.
These are personal design notes from studying how LLM routing systems are built — assembled from public architectures, papers (RouteLLM and friends), and working through the failure modes. Not a description of any system I operate. If you spot something wrong, I'd genuinely like to hear it — email me.