A chat request is one model call. An agent request is a loop, and each iteration re-sends everything that came before.
That difference is not incremental. Teams routinely find that moving a feature from a single call to an agent multiplies token spend by ten or more — and they usually find it on the invoice rather than in design review.
The economics are predictable, though, and mostly fixable without giving up capability.
Where the tokens go
Diagram, as text
- Cost per task leads to Tail cause?
- Tail cause? leads to Truncate tool output, summarise history (bigger iterations)
- Tail cause? leads to Detect no-progress, abort early (more iterations)
- Truncate tool output, summarise history leads to Route cheap steps to a small model
- Detect no-progress, abort early leads to Route cheap steps to a small model
- Route cheap steps to a small model leads to Per-task budget: stop predictably, return partial work
The dominant cost is not what the agent produces. It is what it re-reads.
Each iteration sends the system prompt, the tool definitions, and the entire accumulated history of actions and results. History grows every turn, so the cost of turn n is roughly proportional to n, and total cost for a loop of N turns scales with N² rather than N.
A twenty-step agent does not cost twenty times a single call. It costs closer to two hundred.
| Contributor | Behaviour | Lever |
|---|---|---|
| Conversation history | Grows every turn; re-sent every turn | Compaction, subagents |
| Tool definitions | Fixed, re-sent every turn | Fewer tools; retrieve them |
| Tool results | Often the largest single item | Truncate and summarise at source |
| Thinking tokens | Billed as output, per turn | Route reasoning effort per step |
| Retries | Full-price repeats | Better errors; recover instead of restart |
Tool results deserve particular attention. A tool that returns an entire file, a full API response, or a thousand-row query result pays that cost on every subsequent turn, not just the one that called it. Truncating at the tool boundary is the single highest-leverage cost change in most agents.
Caching is the biggest lever, and loops break it
Prompt caching charges a fraction of the normal rate for a repeated prefix. Agents should be its ideal customer — the same system prompt and tool definitions on every turn.
The catch is that caching works on an exact, contiguous prefix. Change one byte near the start and everything after it misses.
So the ordering rule is: stable content first, volatile content last.
[ system prompt ] stable — cacheable
[ tool definitions ] stable — cacheable
[ compacted history ] changes occasionally
[ recent turns ] changes every turn
[ current observation ] volatileThree habits quietly destroy this: putting a timestamp or request ID in the system prompt, reordering tool definitions between calls, and re-summarising history every turn rather than at a threshold. Each invalidates the prefix and turns a cheap turn into a full-price one.
Route per step, not per application
Choosing one model for the whole agent leaves money on the table, because a loop contains steps of wildly different difficulty.
Deciding which of eight tools to call, extracting a field from a response, or judging whether output looks well-formed are all easy. Designing an approach, debugging a subtle failure, or resolving contradictory evidence are not.
Route accordingly: a small fast model for classification, extraction, routing and formatting; a large model for the genuinely hard steps. The same applies to reasoning effort — a per-step decision, not a global setting.
The constraint is that switching models mid-loop can break your cache prefix, since caches are per model. Group consecutive steps by model where you can, rather than alternating.
Latency is structural
Agent latency is not the model being slow. It is serialisation: each step waits for the previous one, and total time is the sum.
Four things actually help:
Remove steps. The most effective lever by far. A higher-level tool that does in one call what previously took four removes three round trips and three observation costs. This is the same design pressure that improves reliability.
Parallelise reads. Independent lookups should run concurrently. Writes should not.
Stream what you can. Streaming does not reduce latency but changes the experience of it. An agent that shows which step it is on feels responsive; one that shows nothing for forty seconds feels broken, at identical speed.
Cut retries. Every recovered failure is a saved round trip. Good error messages are a latency optimisation.
Compaction pays twice
Compacting history reduces cost on every subsequent turn, not just the current one, because the compacted history is re-sent each time. Compacting at turn ten reduces the cost of turns eleven through fifty.
The same reasoning explains why subagents can be cheaper despite spawning more calls. A subagent that spends thirty thousand tokens exploring and returns a one-thousand-token summary keeps twenty-nine thousand tokens out of the parent's history — where they would otherwise have been re-sent on every remaining turn.
That only holds when the subagent genuinely compresses. One that returns its full transcript adds cost and a lossy handoff for no benefit.
Budget, and fail predictably
Uncapped agents are how a small bug becomes a large invoice. Cap tokens, iterations, wall-clock and spend — per run and per user.
Where the budget is spent matters as much as its size. Reserve enough to exit cleanly: an agent that burns its entire allowance and stops mid-action leaves work in an unknown state, and someone pays twice — once for the failed run, again for the cleanup.
Check your understanding
Common mistakes
Costing an agent like a chat feature. Loops re-send history; spend scales quadratically with turns.
Volatile values in the system prompt. Timestamps and request IDs destroy the cache prefix.
Returning full tool results. You pay for that payload on every remaining turn.
One model for the whole loop. Most steps do not need your best model.
Global reasoning effort. Route it per step like any other cost.
Re-summarising every turn. Compact on a threshold; constant rewriting invalidates the cache.
Alternating models step by step. Each switch abandons a warm cache.
Uncapped loops. The standard way to turn a bug into an invoice.
Optimising cost per call. It reorders your options wrongly.
Practice
An agent costs £0.40 per task on average and £4 at the 95th percentile. Design the cost controls, and say which single change you would make first.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Find out whether the tail is more iterations or bigger iterations.
Show a strong answer
Diagnose before controlling. Split the tail by cause:
- More iterations — the loop is running longer.
- Bigger iterations — context growth per step, usually from large tool outputs.
- Both, which is the common case, because the two multiply.
The single change I would make first: truncate tool outputs before they enter the context, keeping the full output retrievable by reference. A tool returning a 40,000-token file dump is re-sent on every subsequent iteration, so one such call in a 20-iteration loop can account for most of the task's cost. This one change usually collapses the tail and costs almost nothing in capability.
The rest, in order:
- Summarise older iterations past a threshold so context grows linearly rather than quadratically.
- Detect no-progress loops — the same action with the same arguments — and abort early. The budget should not be the thing that stops an oscillation.
- Route per step. Tool selection and argument extraction can use a small model; only planning needs the expensive one. Most iterations are the former.
- Set a per-task budget at roughly 4× the median, with a clear "stopped: budget exhausted" outcome and partial results returned. Predictable failure beats an unbounded bill.
- Cache the invariant prefix, and order the prompt so it survives.
What to publish: cost per successful task at P50 and P95, not the mean. The mean hides the tail, and the tail is where the money goes — as the numbers in this question already show.
Users say the agent 'takes forever'. It completes in 25 seconds and the model calls total 8 seconds. Where is the rest, and what would you do?
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Seventeen seconds is not the model. Account for it before optimising anything.
Show a strong answer
Account for the 17 seconds first, because optimising the model would address less than a third of the problem.
Likely contributors:
- Serial tool calls. Five tools at 2 seconds each, executed one after another, is 10 seconds. Many are independent — three lookups that do not depend on one another can run concurrently.
- Cold starts on a sandbox or container per run.
- Retrieval and reranking, often unmeasured because they are not the model.
- Round trips. Each iteration is a network call to the provider plus tool calls; ten iterations is a lot of fixed latency.
- Waiting for the complete response before showing anything.
What I would do, in order:
- Stream, and show intermediate progress. Much of "takes forever" is silence. Displaying the agent's steps as they happen changes the perceived duration substantially without making anything faster — and it is usually the cheapest win available.
- Parallelise independent tool calls. Requires the loop to support multiple calls per iteration, and it is often the largest real reduction.
- Reduce iterations, which reduces round trips as well as tokens — better tool descriptions and fewer, larger steps.
- Warm the sandbox, keeping a pool ready rather than creating one per run.
- Only then look at the model — a smaller model for the non-planning steps.
The measurement to add: a latency breakdown by stage on every run, so this question is answerable from a dashboard rather than from an investigation. Twenty-five seconds with 8 in the model is a system-design problem, not a model-selection problem, and without the breakdown teams reliably reach for the model first.
Key takeaways
- Agent cost scales with turns squared, because every turn re-sends the accumulated history.
- The biggest single win is usually truncating tool results at source — you pay for them on every later turn.
- Order for cacheability: stable prefix first, volatile content last. Check your cache hit rate before anything else.
- Route per step. Small models for classification, extraction and formatting; large ones for the hard steps.
- Latency is serialisation — remove steps, parallelise reads, and stream progress.
- Compaction and subagents pay repeatedly, by keeping tokens out of every future turn.
- Cap tokens, iterations, time and spend — and reserve budget to exit cleanly.
- Optimise cost per solved task, never cost per call.
Further reading
- Reiner Pope et al., "Efficiently Scaling Transformer Inference" (MLSys 2023).
- Woosuk Kwon et al., "PagedAttention" (SOSP 2023) — memory and batching economics.
- Jeffrey Dean and Luiz André Barroso, "The Tail at Scale" (CACM, 2013) — why P95 dominates perceived quality.
- Anthropic, "Building effective agents" (2024) — fewer, larger steps.
- Google SRE, Site Reliability Engineering (O'Reilly, 2016), chapter 21 — budgets and predictable degradation.