LLM Inference & Serving

You send a prompt and wait three seconds for a reply. Where does that time go?

Answering properly is worth doing, because almost every serving decision — batch size, model size, hardware, price per request — follows from the mechanics of how a model produces text.

Generation is sequential

A model does not compose a sentence and return it. It predicts one token at a time, and each prediction depends on every token before it, including the ones it just produced.

To generate 200 tokens, the model runs 200 times.

This is why output length dominates latency. A request producing 50 tokens is roughly four times faster than one producing 200, regardless of how long the prompt was. The single most effective latency fix in many systems is simply asking for shorter answers.

Two phases with opposite characteristics

Inference splits into two stages that stress the hardware differently, and confusing them leads to bad capacity decisions.

Prefill processes your entire prompt. Every token can be handled in parallel, so this stage saturates the GPU's arithmetic units. It is compute-bound, and it produces the delay before the first word appears.

Decode generates the output, one token per step. Each step does comparatively little arithmetic but must read the model's weights from memory to do it. The GPU spends most of its time waiting on memory rather than calculating, making this stage memory-bandwidth-bound.

The practical upshot: a long prompt costs you once, a long answer costs you every token. And because decode is memory-bound, a GPU generating for a single user is mostly idle — which is precisely the gap batching exists to fill.

The KV cache

Here is the optimisation that makes generation practical, and the one that consumes your memory.

Naively, predicting token 201 means reprocessing tokens 1–200 from scratch. That work is identical every step, so it is cached: the intermediate values for each previous token — the keys and values — are kept and reused. Each new step only processes the newest token against that cache.

Without it, generation cost would grow quadratically with length. With it, each step is roughly constant work.

The cost is memory, and it is substantial. The cache grows with sequence length × number of concurrent requests, and on a busy server it can consume more GPU memory than the model weights themselves.

That single fact explains most serving behaviour:

• Long contexts are expensive in memory, not just in tokens billed.
• Concurrency is capped by memory, not by compute — you run out of room for caches before you run out of arithmetic.
• Memory fragmentation matters, because reserving worst-case space per request wastes most of it. Modern servers allocate the cache in small fixed-size blocks on demand, which lets far more requests share a GPU.

Batching, and why the naive form fails

GPUs are enormously parallel, and a single decode step for one user uses a fraction of that capacity. Processing several requests together costs barely more than processing one — so batching is where throughput comes from.

Static batching collects requests, runs them together, and returns when all are done. The flaw is obvious once you picture it: requests in a batch finish at different times, because output lengths differ. A batch of eight where one request needs 500 tokens and the rest need 20 leaves seven slots idle for the remainder of the run.

Continuous batching fixes this by working at the level of individual decode steps rather than whole requests. When any request finishes, a waiting request takes its place immediately. The batch is constantly refilled rather than drained and refilled.

This is the highest-leverage optimisation in LLM serving, and it is why you should use a purpose-built inference server rather than calling a model in a loop. The difference in requests served per GPU is large — several-fold, not a few percent.

Making the model smaller

Quantization stores weights at lower numerical precision — 8-bit or 4-bit instead of 16-bit. The model shrinks proportionally, which produces three compounding benefits: it fits on cheaper hardware, less memory traffic means faster decode, and freed memory allows larger batches.

The cost is some accuracy. Moving to 8-bit is usually close to lossless; 4-bit is noticeable on harder reasoning tasks and fine for many others. Measure it on your own evaluation set rather than trusting a general benchmark — this is a decision your evals should make, not a blog post.

Distillation trains a small model to imitate a larger one, often achieving most of the quality at a fraction of the cost for a narrow task.

Speculative decoding uses a small fast model to guess several tokens ahead, then has the large model verify them in one pass. Correct guesses are kept, wrong ones discarded. Because verification is parallel and cheap, accepted guesses come nearly free. Output is identical to what the large model would have produced — you are spending idle compute to skip memory-bound steps.

Routing to the right model

The most reliable cost lever is not making one model faster. It is not using the expensive model for easy requests.

Most production systems run several tiers and route between them: a small model for classification, extraction and simple queries; a large one for genuine reasoning. A classifier or a set of rules decides which. Escalate when the small model signals low confidence.

Since capable small models are dramatically cheaper per token than frontier models, moving even half your traffic down a tier changes your bill more than any inference tuning will.

Build or buy

API providers — no infrastructure, immediate access to the best models, pay per token. You accept rate limits, data leaving your network, and prices and deprecations you do not control.

Self-hosting — fixed GPU cost regardless of usage, full control over data and versions, and real operational burden. It becomes cheaper only at sustained high volume, because an idle GPU still bills.

The honest guidance: start with an API. Self-host when volume is predictable and large, when data residency requires it, or when you need a model you cannot get otherwise. Building GPU serving infrastructure before you have proven the product is a costly detour.

Check your understanding

Common mistakes

Optimising prompt length while ignoring output length. Decode dominates; a shorter prompt saves one pass, a shorter answer saves hundreds.

Serving with a plain inference loop. Without continuous batching you may be using a small fraction of the GPU you are paying for.

Sizing capacity by request rate alone. Concurrency is bounded by KV cache memory, so average sequence length matters as much as requests per second.

One model for everything. Sending a classification task to a frontier model is the most common source of avoidable spend.

Self-hosting too early. Idle GPUs bill continuously, and the operational load is real.

Trusting published benchmarks for quantization. Accuracy loss is task-specific. Measure on your own evaluations.

Practice

Key takeaways

• Generation is sequential, so output length drives latency far more than prompt length.
• Prefill is compute-bound, decode is memory-bound — measure time-to-first-token and inter-token latency separately.
• The KV cache makes generation linear rather than quadratic, and its memory use is what actually caps concurrency.
• Continuous batching refills slots as requests finish and is the single biggest throughput win in serving.
• Batch size trades throughput against per-request speed — serve latency-sensitive and bulk workloads from separate pools.
• Quantization, distillation and speculative decoding shrink cost; validate quality on your own eval set.
• Model routing beats micro-optimisation — the cheapest request is the one a small model handled.
• Start with an API; self-host only for sustained volume, data residency, or model control.

Further reading

• Reiner Pope et al., "Efficiently Scaling Transformer Inference" (MLSys 2023) — the prefill/decode analysis.
• Woosuk Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention" (SOSP 2023) — paged KV cache.
• Noam Shazeer, "Fast Transformer Decoding: One Write-Head is All You Need" (arXiv, 2019) — multi-query attention and its memory effect.
• Joshua Ainslie et al., "GQA: Training Generalized Multi-Query Transformer Models" (EMNLP 2023).
• Tri Dao et al., "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness" (NeurIPS 2022).