The model held up better than expected. The part that broke was the piece everyone assumed was the safety net.
One of the systems I work on runs an LLM audit on user submissions: a user submits, a service kicks off a model review in the background, and the result comes back to decide what happens next. It's a design I'd have described as "sensibly conservative" — asynchronous, decoupled, with a human fallback for anything the automation can't finish in time.
Then traffic jumped roughly 52x, and I learned which parts of "sensibly conservative" were real and which were assumptions wearing a hard hat. Written up here in general terms, because the lessons transfer.
Service A hands the audit task to a backend service that owns a queue and a worker pool; workers call the model; results flow back to service A asynchronously over Kafka. If service A doesn't hear back within N minutes, the case falls through to an ops queue for manual human review.
Each choice is defensible. Async with a queue means submission latency doesn't depend on model latency. The timeout guarantees no case waits forever. The human fallback guarantees no case is decided by silence. At normal traffic, all of this was true.
Here's the thing: the model didn't fall over, and neither did the queue. Queues are built to absorb bursts — that's their job. What happened instead was quieter and worse. The queue absorbed the burst, wait times stretched past the timeout, and the timeout did exactly what it was configured to do: it routed the overflow to the ops queue.
At 1x traffic, timeout cases were a trickle a small team handled comfortably. At 52x, the "fallback" was receiving more volume than the automated path was ever designed for — and the consumers of that queue were people. You can scale a worker pool in minutes. You cannot 52x a team of human reviewers, on any timeline, at any budget.
A fallback that routes to humans isn't a fallback under load — it's a buffer whose consumers don't autoscale. If the fallback can't absorb your worst day, what you actually have is a normal-day design with a comforting label on it.
Under pressure, the goal clarified into one sentence: maximize the share of traffic the automated path completes, because every case automation finishes is a case ops never sees. Three levers, in increasing order of impact:
1. Raise the timeout. The timeout was tuned for normal-load wait times, so under load it was declaring "too slow" on cases the pipeline would have finished moments later — converting temporary queue delay into permanent human work. Raising it traded per-case latency for automation rate, and during a spike that's the right trade. The realization that stuck with me: a timeout is not a technical constant, it's a policy decision about who does the work — and it had been set without anyone asking that question at 52x.
The follow-on conclusion: if spikes are regular or periodic — and for many products they are, driven by campaigns, market events, or time-of-day patterns — a static timeout is the wrong tool entirely. Better to make it dynamic: derive it from observed conditions (current queue depth, rolling p95 processing time) so the system widens its own patience under load and tightens it when things are calm. The latency-vs-automation trade still gets made — it just gets made continuously by the system, instead of once by whoever happened to configure the constant, at whatever traffic level happened to exist that day.
2. Increase model-side batching. The model service could process substantially more tasks per cycle than it was configured for — the conservative batch size was another normal-load assumption. Pushing it to roughly 300 tasks per round traded a bit of per-task latency for a large jump in throughput. Same trade as the timeout, made at a different layer.
3. Scale out the workers — but threads, not instances. The worker pool went from 15 threads × 4 instances (60 concurrent tasks) to 45 threads × 4 instances (~180 concurrent) — 3x the concurrency without adding a single machine. Why not just add instances, the reflexive answer? Because the workers spend almost all their time waiting: an LLM call is long-latency I/O, so a thread sits idle for seconds while the model thinks. CPU and memory per instance were nowhere near saturated — the instances weren't the bottleneck, the thread count was. Tripling threads on existing machines was free capacity, available in a config change; new instances would have cost money, taken longer to provision, and solved a problem we didn't have.
The step that must come before turning that dial: check the model's TPM (tokens-per-minute) headroom. Concurrency past what the model can absorb doesn't add throughput — it just relocates the queue from your side to the provider's, and buys you rate-limit errors. The useful ceiling is roughly:
max useful concurrency ≈ (TPM × avg task duration in minutes) / avg tokens per task
Work out that number first, then size the pool to approach it — not exceed it. In our case, 180 concurrent sat comfortably within headroom; if it hadn't, no amount of thread tuning would have helped, and the fix would have been a quota increase or a second model deployment instead.
The spike left a residue: everything that had already fallen into the ops queue. The insight that mattered here is that most of those cases weren't things automation couldn't handle — they were things automation hadn't gotten to in time. So instead of asking humans to grind through them, we re-ran them through the model pipeline.
The one hard constraint: the replay must not degrade live traffic. New submissions from real users waiting right now always outrank week-old backlog. That means the backlog re-run gets treated as a separate, lower-priority lane — fed into the pipeline at a controlled rate, using headroom, never competing head-to-head with the real-time path. Two lanes, one pipeline, strict priority. Ops then handles only what fails automation twice — which is a workload a human team can actually absorb.
The real output of an incident isn't the fix — it's the runbook. Distilled, ours looks like this:
Reading order matters: step zero is always the infra check — TPM headroom — because every lever downstream is capped by what the model can absorb. Then the three levers together push automation rate back up; the two-lane split recovers the backlog without touching live users; and the ops queue shrinks back to its intended role — genuine edge cases, not overflow.
Size the fallback for the failure day, not the normal day. The whole point of a fallback is to be there when things go wrong — which is precisely when volume is abnormal. Ours was implicitly sized for normal-day overflow, and that assumption was invisible until the day it mattered.
Interrogate every "conservative" constant — then ask if it should be a constant at all. The timeout, the batch size, the pool size — each was individually reasonable and collectively an unexamined bet on traffic staying normal. Load testing would have surfaced them; instead, production did. And where the load pattern is recurring, the right fix isn't a better constant, it's replacing the constant with a feedback loop.
Automation rate is the metric that matters for human-in-the-loop systems. Not throughput, not latency — the fraction of cases that never reach a human. Every lever above is really that one metric wearing different clothes.
Replays are second-class traffic, always. Any recovery process that competes with live users has just extended the incident by other means.
The system now handles far more than it did — but the durable change is in how I read designs. When someone shows me an architecture and points at the fallback path, my first question is no longer "does it exist?" It's "what happens to it at 52x?"
Incident described in general terms — numbers rounded, identifying details omitted. The lessons are the point.