How to Reduce Token Costs in an LLM Product at Scale

"How would you reduce token costs?" has become a standard system design question, and it has a standard weak answer: use a smaller model. That is a real lever, but it is roughly third on the list, and leading with it suggests you have not thought about where the money actually goes.

Here is the ordering that holds in practice, strongest first.

1. Do not call the model

The cheapest inference is inference that does not happen.

Prompts repeat far more than people expect — support questions, retried requests, templated flows, popular queries. A cache in front of the model converts a repeat into a lookup. Not a cheaper call: no call. If a third of your traffic repeats, you have removed a third of your inference spend, and you have removed it entirely rather than discounting it.

Nothing else on this list is as strong, because everything else makes calls cheaper while this one deletes them.

2. Call a cheaper model when a cheaper model will do

Most requests hitting a general-purpose LLM product do not need the frontier model. Classification, extraction, short factual answers, routing decisions — a small model handles these fine, at a fraction of the price per token.

A router in front of the model tiers, sending the easy fraction to the cheap tier and escalating only what needs it, is the second-biggest lever. If 70% of traffic can be served by a model that costs a fifth as much, your blended cost drops by more than half without a single user noticing.

The interesting part of this answer, and the part interviewers listen for, is how you decide. Confidence thresholds, request classification, escalate-on-failure — all defensible. "I'd use heuristics" is not.

3. Put less in the prompt

Now we are at the token level, and this is where RAG systems bleed.

Prompt tokens are charged on every request. A retrieval step that stuffs twenty passages into context instead of the five that matter is paying four times over, forever, for recall it did not need. Rerank and truncate. Trim system prompts that grew by accretion. Summarise conversation history rather than replaying it.

This is unglamorous and it is frequently the largest single line item in a mature RAG product.

4. Then, and only then, tune the model

Smaller models, quantisation, distillation, fine-tuning a small model to match a large one on your specific task. All real, all effective, all significantly more work than the three levers above — and all of them get less impressive if you have not already stopped making unnecessary calls with oversized prompts.

The gateway is where you implement all of this

The reason these levers tend to arrive together in a design is that they all live in the same place: a gateway between your services and the model providers. Caching, routing, per-tenant limits, prompt screening, and the measurement that tells you whether any of it is working.

It is also the natural place to enforce budget caps, which matters more than it sounds. An agent loop with no ceiling will happily spend a thousand dollars retrying a task nobody is waiting for any more. Bounding that is a design requirement, not an optimisation.

What a strong answer sounds like

"First I'd cache — a meaningful share of requests repeat, and a hit removes the call entirely rather than making it cheaper. Then I'd route: most requests don't need the frontier model, so a cheap tier with escalation cuts blended cost substantially. Then I'd attack prompt size, because in a RAG system the retrieved context is usually the largest term and reranking down is nearly free. Model-level work comes after that. And I'd put all of it behind a gateway so it's enforceable and measurable, with a hard budget ceiling per tenant."

That is four levers, ordered by impact, with a reason attached to each. It takes twenty seconds and it is a completely different answer from "use a smaller model."

---

The ordering above is arguable in the abstract and obvious once you can see the bill. Build a gateway, route traffic through cheap and expensive tiers, turn caching on and off, and watch which change actually moves the number.