For most of the deep-learning era, a model's capability was fixed at training time. You made models better by training bigger ones on more data, and at inference you simply ran them.
Reasoning models broke that assumption. They are trained to produce an extended internal chain of thought before their final answer, and their accuracy improves as they are allowed to think for longer. Capability became something you can buy at request time, by spending more tokens.
That is a genuinely new lever, and it changes system design: quality is no longer a fixed property of the model you chose, it is a per-request decision with a price.
A second scaling axis
There are now two ways to spend compute for quality:
| Axis | When you pay | What you get | Who controls it |
|---|---|---|---|
| Training compute | Once, up front | A better base model | The model provider |
| Test-time compute | Every request | More thinking on this request | You |
The second row is the one you can act on. It also means the interesting question shifted from "which model is smartest?" to "how much thinking is this particular request worth?"
The knob you actually turn
In practice you rarely tune anything as exotic as a search strategy. You set a thinking budget — variously exposed as a reasoning-effort level or a maximum thinking-token count — and the model decides how to spend it.
Treat it as a per-request routing decision, not a global setting:
- Off or minimal — retrieval, summarisation, classification, formatting, extraction, chat. Anything where the answer is in the context and just needs restating.
- Moderate — multi-step tool use, ambiguous requirements, code that must be right the first time.
- High — genuine problem solving: tricky debugging, planning under constraints, mathematics, design trade-offs with no obvious answer.
The mistake is setting one level for the whole application. A support assistant that answers "what is my plan?" with high reasoning effort is paying premium prices for a database lookup.
Where reasoning earns its cost
The evidence is consistent about the shape of the benefit, even where exact numbers vary by model and benchmark.
It helps most where the task decomposes into steps that must each be right: mathematics, competitive programming, multi-constraint planning, debugging from symptoms, and anything requiring the model to notice its own mistake and back up.
It helps little where the work is recall or transformation. If the answer is already in the context, thinking about it longer does not make it more present. Retrieval-grounded question answering, summarisation and format conversion see modest gains at best.
It can actively hurt in two ways worth naming. Latency-sensitive interactions become unusable when the first useful token arrives seconds late. And on simple tasks, extended deliberation can talk a model out of a correct initial answer — the overthinking failure, where more reasoning produces a worse result than none.
Sequential and parallel are different purchases
Diagram, as text
- Problem leads to Which kind of compute?
- Which kind of compute? leads to Sequential: longer reasoning (refine one attempt)
- Which kind of compute? leads to Parallel: sample N (many attempts)
- Parallel: sample N leads to Can you tell which is right?
- Can you tell which is right? leads to Select best (yes: tests, verifier, majority)
- Can you tell which is right? leads to Wasted spend (no)
- Sequential: longer reasoning leads to Answer
- Select best leads to Answer
There are two ways to spend more at inference, and they behave differently.
Sequential — let one attempt think for longer. Cheaper, simpler, and the default. Returns diminish: doubling the budget does not double accuracy, and every task has a point past which more thinking adds cost and nothing else.
Parallel — sample several independent attempts and select among them, by majority vote or by a verifier. This buys something sequential cannot: decorrelated errors. Where a single long chain can commit to a wrong path early and rationalise the rest of the way, independent attempts fail differently, and agreement between them is genuine evidence.
Parallel costs multiples of a single call and needs a selection mechanism. Majority voting only works when answers are comparable — fine for a numeric result, meaningless for an essay. Where you have a deterministic check available, verifier-based selection is strictly better than voting.
Reasoning inside an agent loop
For agents, the important capability is not thinking once at the start but thinking between tool calls — reading a result, reconsidering the plan, and choosing the next action deliberately.
This changes loop economics. Every iteration now carries a thinking cost on top of the tool call, so loops that were affordable become expensive. It also changes where reasoning pays: deciding which tool to call and whether the result was what was expected is exactly the kind of multi-step judgement that reasoning improves, which is why agentic workloads benefit from it more reliably than chat does.
Hidden reasoning and what it costs you
Providers generally do not return raw reasoning traces — you get a summary, or nothing. That is a deliberate product decision, and it has real engineering consequences.
You are debugging a process you cannot fully see. When an agent takes an inexplicable action, the deliberation that produced it may not be available to you. Plan observability around inputs, tool calls and outputs, which you do control, rather than assuming you can inspect the model's reasoning after the fact.
Reasoning is not the answer to everything
Two honest limits.
Reasoning improves the process, not the facts. A model reasoning carefully from a wrong retrieved document produces a well-argued wrong answer. Reasoning is not a substitute for grounding, and it can make hallucinations more persuasive rather than less frequent.
And benchmark scores flatter it. Reasoning models are evaluated on exactly the tasks they are strongest at, usually without a cost budget. A model that wins on a maths benchmark by spending twenty times the tokens has not necessarily won anything you care about.
Check your understanding
Common mistakes
One reasoning level for the whole application. Route it per request; most requests do not need it.
Not reserving output space above the thinking budget. The model reasons to the limit and gets truncated before answering.
Comparing cost per call instead of cost per solved task. The expensive model that succeeds first time is often the cheap one.
Using reasoning to fix grounding problems. Careful reasoning over a wrong document is a better-argued wrong answer.
Streaming a reasoning model into a latency-sensitive UI without a plan. Users watch nothing happen for several seconds. Show that thinking is in progress, or route to a fast model.
Assuming more thinking is monotonically better. On simple tasks it can reverse a correct answer.
Treating benchmark wins as product wins. Almost all published reasoning results are reported without a cost budget.
Practice
You must decide whether to enable extended reasoning for a code-review assistant. Design the experiment that answers it, and say what result would make you say no.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Cost and latency are certain; the quality gain is the thing to measure.
Show a strong answer
Build the evaluation set first, because without it the decision is aesthetic. Fifty to a hundred real review cases with known outcomes: bugs a human found, changes that were fine, and — importantly — cases where a reviewer flagged something wrongly, so you can measure false positives as well as catches.
Run three arms on the same set: no extended reasoning, moderate, and high. Hold everything else constant.
Measure four things:
- Catch rate on the genuine defects.
- False positive rate, which is what determines whether developers keep using it. A reviewer that flags noise gets muted, and then catches nothing.
- Cost per review, in tokens.
- P95 latency, because a review that arrives after the pull request is merged is worth nothing regardless of quality.
What would make me say no:
- A catch-rate improvement inside the noise of the sample size. With 100 cases, a two-point difference is not a result.
- Catch rate up but false positives up proportionally. That is not better judgement, it is a lower threshold — obtainable more cheaply by prompting.
- Latency crossing the workflow boundary. If reviews must land before a developer context-switches away, a jump from 20 s to 3 minutes fails regardless of accuracy.
- Cost per review exceeding what the catch is worth. Straightforward arithmetic: cost per review × reviews per month, against the cost of a defect reaching production.
The likely nuanced outcome: reasoning helps on a subset — concurrency bugs, subtle logic errors — and does nothing for style and convention. That argues for routing rather than a global switch: cheap pass on everything, escalate to reasoning on diffs touching risky areas.
A model provider hides the reasoning tokens but bills for them. What problems does this create, and how would you operate around it?
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Consider debugging, cost control and evaluation separately.
Show a strong answer
Three distinct problems:
- You cannot debug the failure. When the answer is wrong, the reasoning is where the mistake happened, and it is unavailable. You are left inferring the cause from the conclusion, which is guesswork.
- Cost becomes unpredictable per request. Reasoning length varies with problem difficulty, so two superficially similar requests can differ several-fold in cost with no visible reason. Budgeting from average input and output length is wrong.
- Evaluation gets coarser. You can score the final answer but cannot tell whether a correct answer came from correct reasoning or from a lucky guess — which matters, because the second does not generalise to the next case.
How I would operate around it:
- Meter and cap by reasoning tokens explicitly, not just by output tokens. Set a per-request ceiling so a pathological case cannot consume a hundred times the budget, and alert on the distribution rather than the mean.
- Track cost per successful outcome, not per call. That is the number that makes reasoning's value visible: a higher per-call cost with a lower retry rate can be cheaper overall.
- Evaluate on outcomes you can verify independently — does the generated code pass the tests, does the extracted figure match the source — so hidden reasoning does not block measurement even though it blocks inspection.
- Log everything you do control: full input, final output, token counts, latency, and the request's identity. When the reasoning is unavailable, the surrounding context is the only reconstruction you have.
- Keep a fallback path to a non-reasoning model, so a cost or latency problem has an immediate lever that does not require a provider change.
Key takeaways
- Reasoning models add a second scaling axis: capability you buy per request rather than at training time.
- The practical control is a thinking budget, and it belongs per request — most traffic does not need it.
- It pays on multi-step reasoning, planning and agentic tool use; it does little for retrieval, summarisation and formatting.
- Thinking tokens are billed output tokens — they cost output rates, add latency, and will truncate your answer if you do not reserve room.
- Sequential buys depth cheaply; parallel buys decorrelated errors and needs a selection mechanism.
- Inside agents, the value is thinking between tool calls — and that makes every loop iteration more expensive.
- Judge on cost per solved task, including retries and human correction, not cost per call.
- Reasoning improves the process, not the facts — it is not a substitute for grounding.
Further reading
- Jason Wei et al., "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" (NeurIPS 2022).
- Charlie Snell et al., "Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters" (arXiv, 2024).
- Xuezhi Wang et al., "Self-Consistency Improves Chain of Thought Reasoning in Language Models" (ICLR 2023) — majority selection over parallel samples.
- Hunter Lightman et al., "Let's Verify Step by Step" (ICLR 2024) — process supervision and verifiers.
- Shunyu Yao et al., "Tree of Thoughts: Deliberate Problem Solving with Large Language Models" (NeurIPS 2023).