Designing for a Workload That Thinks: Capacity Planning When Cost Is Per Token

The first time a model-backed feature costs real money, it usually isn't traffic. It's a retry.

Something upstream starts timing out, the client library does what client libraries do, and the same generation is paid for two or three times over — each attempt abandoned just before it would have finished. The graph that shows the problem isn't CPU or connections. It's the bill.

That's a small incident with a large implication. Every capacity technique most of us learned assumes retries are cheap, service time is roughly stable, and adding servers adds capacity. A workload that thinks breaks all three.

Service time is a distribution, not a number

A conventional web handler's latency clusters. The slow path might be a few times the fast path, and you plan against a percentile without thinking hard about the shape.

Generation time is roughly proportional to how much output the model produces, and the same endpoint can produce a one-line confirmation or several pages of structured text depending on what the user asked. That's an order-of-magnitude spread inside a single route, with no error and no warning.

Queueing behaviour is sensitive to exactly this. Waiting time grows with the variability of service time, not just its mean — so the comfortable utilisation target you inherited from CPU-bound services is too aggressive here. At the same average load, higher variance means longer queues, and the tail gets worse faster than intuition suggests. The practical consequence: you must run this workload at lower utilisation than your other services, and the instinct to "get the fleet efficient" is actively harmful.

What you are actually rationing

Not CPU. Not connections. Two things.

Concurrency, because whether you call a provider or serve your own models, there is a ceiling on simultaneous in-flight generations, and it doesn't move because you added web servers. If you self-host, the ceiling is a memory and batching property of the serving tier; if you don't, it's an allocation. Either way it's largely fixed on the timescale of an incident.

Tokens, because that's what the meter counts. Cost scales with the volume of text moved through the model, not with the number of requests. Two facts follow that are worth stating plainly: a request is not a unit of consumption, and you cannot cap your spend by capping requests per second. A rate limit expressed in requests permits an unbounded bill.

Retry stopped being free

A classical retry costs a socket and some cycles. Retrying a generation that had already produced most of its output costs the whole generation again — and retrying on timeout is the pathological case, because a timeout is most likely on exactly the long, expensive requests, so you abandon the most costly work and immediately reorder it.

Three rules follow. Don't retry a timeout with identical parameters; if you retry at all, retry smaller — a tighter output cap, less retrieved context, a cheaper model. Make retries draw from the same budget as the original attempt, so three attempts can't cost three times the ceiling. And put a spend ceiling on the request and on the tenant, because the failure mode you actually fear is one caller in a loop, not general growth.

Admission control, because you can't scale out of it

When capacity is a mostly fixed concurrency allocation, autoscaling is not the answer to overload — the extra instances queue against the same ceiling and turn a capacity problem into a distributed one. The queue in front of that ceiling is the design.

Which means bounded queues and fast rejection, separated by workload class. Interactive requests where somebody is watching a cursor blink, and background requests where nothing is watching, must not share a queue: mixing them lets a batch summarisation job push a user-facing response past the point of usefulness. Give them separate concurrency allocations rather than separate priorities in one pool, so the background class can be starved without any coordination.

The awkward part is that admission control wants a service-time estimate and you don't have one. You can approximate it from what you do know at admission: input size, the requested output cap, model tier, and whether the request is a single call or a loop. That estimate is rough, and rough is enough — it lets you reserve proportionally and reject work you clearly cannot serve inside its deadline, which is most of the benefit.

Cost belongs in the SLO

We're used to reliability and latency objectives. This workload needs a third: cost per successful outcome — per resolved ticket, per document processed, per answer accepted by the user.

The reason isn't accounting. It's that cost is now tradeable against quality and latency in the same design decision. A smaller model, a shorter context window, fewer retrieved passages, a tighter output cap, an aggressive cache on repeated prefixes — each moves all three dials at once. Without a cost objective you can't tell whether a change was an improvement, because you're optimising two of the three variables and letting the third float.

Stating it as an objective also forces the question of what happens when a budget is exhausted, which should be answered at design time rather than at three in the morning. Degrade deliberately: fall back to a cheaper model, serve a cached answer, disable tool use, or refuse honestly. Any of those beats the default behaviour, which is to keep spending until someone notices.

Latency objectives need a different shape

A single percentile over a bimodal distribution describes nothing that exists. Split the objective by workload class, and set each one against what the user is actually waiting for.

And measure time-to-first-token separately from time-to-completion, because with streaming they're different products. A system tuned for total completion time can feel markedly worse than one that starts producing sooner and finishes later, and if you only track completion you will optimise in the direction that feels worse.

The shape of the answer

None of the mechanisms here are new. Bounded queues, class separation, admission control, budgets, deliberate degradation — this is the same toolkit that protects any saturating resource, and the reason it feels unfamiliar is that we usually get to skip it by adding machines.

You can't skip it here. Plan capacity in concurrency and tokens rather than requests. Plan overload as queueing and degradation rather than scaling. And put a ceiling on everything, because the one resource that never pushes back on its own is the one you're billed for.