LLM Inference at Scale: Why Your Serving Layer Saturates So Fast

There is a number that reframes this whole problem once you internalise it: generation is sequential. A model cannot produce the second token before it has produced the first. No amount of parallelism inside a single request changes that, because the dependency is in the algorithm, not the implementation.

The consequence is that a GPU serving a language model saturates at a request rate that would not even register on a web server. An API server handles thousands of requests a second. A single inference node handles a number you can count on your hands.

Almost everything interesting about designing a serving layer follows from that.

Batching is the throughput lever, not headcount

The instinct when a tier saturates is to add instances. For inference that is the most expensive possible answer, because idle GPU capacity is the most expensive idle capacity you will ever provision — roughly an order of magnitude more per hour than the web server sitting in front of it.

The cheaper answer is batching. Sixteen requests through one forward pass cost roughly what one costs, because the expensive part is moving weights, not doing arithmetic on your particular tokens. So a serving layer that batches gets most of an order of magnitude of throughput out of hardware it already has.

This is why "what's your batch size?" is a real interview question and "I'd autoscale the GPUs" is a weak answer. Autoscaling GPUs is slow — boot times are minutes, not seconds — and expensive, and it is the thing you do after you have made each GPU efficient, not instead.

Buffering, because batching needs something to batch

Batching only works if requests arrive close enough together to be batched. That means a queue between the entry point and the GPUs.

This is the same argument as any spike-absorption design, with one twist: for a web tier a queue is about surviving bursts, and for inference it is also about creating them. You want a small amount of queueing even in steady state, because a queue that is always empty is a serving layer running batches of one.

That trade-off — a little latency in exchange for a lot of throughput — is the core tuning decision, and being able to state it clearly is most of what an interviewer is listening for.

Caching, because the cheapest forward pass is the one you skip

Prompts repeat more than people expect. Templated system prompts, retried requests, popular queries, evaluation runs hammering the same fixtures. A cache in front of the model does not make inference faster; it means inference does not happen.

Given the relative costs — a cache costs about what a Redis costs, a GPU costs 25× a web server — this is the highest-leverage component in the design by a wide margin, and it is one line on a diagram.

What breaks first, and how you would know

The failure mode worth designing for is not a GPU dying. It is a GPU getting slow.

A dead node fails fast, the load balancer notices, traffic routes around it. A node that is merely slow keeps accepting work it cannot finish, so requests pile up behind it, callers block waiting, and the pressure travels back into services that are themselves perfectly healthy. Designs survive the first case and collapse under the second constantly.

If you can name that distinction in an interview — and say what you would do about it, which is timeouts and circuit breakers and shedding rather than more capacity — you are ahead of most candidates.

A serviceable answer

• Batch, and say what size and why
• Queue between arrival and the GPUs, and be explicit that this trades latency for throughput
• Cache exact repeats, because that is the only lever that removes work rather than speeding it up
• Design for degradation, not just death
• Know that GPU hours and tokens are the bill, and that everything else on your diagram is noise by comparison

---

The thing that makes this concrete is watching a serving layer saturate. Build one, push traffic through it until it sheds, then add a cache and watch both the drop rate and the monthly cost fall.