In a conventional system, caching is a performance optimisation. In an AI system it is primarily a cost control, and the numbers are stark enough to change architecture.
A support assistant handling 100,000 questions a day, where a third are effectively the same handful of questions, is paying full price three times over for answers it already produced.
The caching lesson in the high-level module still applies — cache-aside, TTLs, invalidation, stampedes. This lesson covers what is different when the thing you are caching was generated rather than stored.
Three layers, cheapest first
Diagram, as text
- Request leads to Prefix cache: identical system prompt
- Prefix cache: identical system prompt leads to Exact match on normalised query?
- Exact match on normalised query? leads to Serve stored answer (hit)
- Exact match on normalised query? leads to Semantic match above threshold? (miss)
- Semantic match above threshold? leads to Serve stored answer (hit, same permission scope)
- Semantic match above threshold? leads to Call the model (miss)
- Call the model leads to Store with scope in the key
Exact caching
Hash the full request — prompt, model, and every parameter — and store the response.
const key = sha256(JSON.stringify({ model, messages, temperature, tools }));
const hit = await cache.get(key);
if (hit) return hit;Trivially correct, because identical input means identical expected output. It works better than intuition suggests: automated pipelines, retries, shared links and popular questions produce far more exact repeats than people expect.
Two rules make it safe:
Every parameter belongs in the key. Temperature, model version and system prompt all change the output. A key omitting the model returns a small model's answer to a request for the large one.
Never share a cache across users without care. If the prompt contains user data, the key must include the user or tenant. This is the same leak as the CDN lesson, with worse consequences — you would be serving one customer's information to another.
Prefix caching
A subtlety specific to models, and one that is easy to miss.
Most prompts share a long, identical beginning: the system prompt, tool definitions, few-shot examples. Only the user's question at the end differs.
Recall from the serving lesson that prefill processes the prompt and builds a KV cache. If the first 2,000 tokens are identical to the last request, that work is repeatable — providers and inference servers can reuse the cached state and only process the new suffix.
The engineering consequence is concrete: put stable content first and variable content last.
[ system prompt ][ tool definitions ][ examples ] <- stable, cacheable prefix
[ retrieved context ][ user question ] <- varies per requestReordering a prompt so the variable part sits at the front silently destroys this. It is a real and common performance regression that shows up as unchanged behaviour and a higher bill.
Semantic caching
The interesting one. Two users ask:
"How do I reset my password?" "I forgot my password, how can I change it?"
Different strings, identical intent. Exact caching misses; semantic caching does not.
Embed the incoming question, search the cache of previous questions, and if the nearest is above a similarity threshold, return its stored answer.
const v = await embed(question);
const [near] = await cache.search(v, { limit: 1 });
if (near && near.score > THRESHOLD) return near.answer;Hit rates on customer-facing assistants are often substantial, because real user questions cluster heavily — the same skew that makes ordinary caching work.
Three refinements make it safer:
- Segment the cache by user, tenant, language and any filter that changes what a correct answer looks like.
- Never semantically cache personalised answers. "What is my account balance" must never hit another user's entry — and near-identical phrasing makes that failure likely, not hypothetical.
- Log hits with their similarity scores so you can audit what was served and where the threshold sits in practice.
What to cache, and for how long
| Content | Cache? | Notes |
|---|---|---|
| Answers from static documentation | Yes | Long TTL; invalidate when the source changes |
| Embeddings of documents | Yes | Effectively permanent per model version |
| Classification and extraction | Yes | Deterministic-ish, high repeat rate |
| Personalised answers | Only per user | Never share across users |
| Anything time-sensitive | Short TTL | "What is today's status" ages in minutes |
| Agent tool results | Depends | Reads yes, actions never |
TTL is the same reasoning as the consistency lesson: how stale may this be? An answer built from documentation that changes weekly can live for hours. An answer about current system status cannot.
Crucially, cached answers must be invalidated when their source documents change, or your assistant will confidently quote a policy you replaced last month.
Streaming complicates things
Most AI products stream. That interacts with caching in two ways worth planning for.
You must store the complete response as it streams, not only send it, or you have nothing to cache.
And a cache hit returns instantly, which is a jarringly different experience from a token-by-token stream. Many products deliberately replay cached answers with simulated streaming so the interaction stays consistent.
Measuring whether it works
Track three numbers together:
- Hit rate, split by layer — exact, prefix, semantic. They have very different economics.
- Cost saved, computed as hits multiplied by what those calls would have cost. This is the number that justifies the work.
- Wrong-hit rate for semantic caching, measured on a labelled sample. Without it you are trading correctness for cost blindly.
Check your understanding
Common mistakes
Omitting parameters from the cache key. Model version and temperature change the output.
Sharing a semantic cache across users. The most serious failure available here.
Reordering prompts so the variable part comes first. Silently destroys prefix reuse; nothing breaks, the bill rises.
Choosing a threshold to maximise hit rate. Optimising the wrong metric — measure wrong-hits.
Never invalidating on source change. Confidently quoting superseded policy.
Not storing streamed responses. Streaming without accumulation means nothing to cache.
Practice
Design caching for a customer support assistant answering questions about a product, where some answers depend on the specific customer's plan and region. State each layer and what must never be cached.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Separate what varies by user from what does not, before choosing thresholds.
Show a strong answer
Split the request into parts that vary and parts that do not — the design follows from that split.
- Prefix cache the system prompt and tool definitions. Identical for every request, large, and safe. Order the prompt so the invariant part comes first, since prefix caching only helps up to the first difference — putting the customer's name at the top defeats it entirely.
- Cache retrieval results by query, keyed on the normalised question plus the permission scope. Retrieval is deterministic given the corpus version, so include a corpus version in the key and invalidate on reindex.
- Semantic cache only the general-knowledge answers — "how do I reset my password", "what are your support hours" — with the customer's plan and region in the cache key, not merely in the prompt. Two customers on different plans must never share an entry, and the safest way to guarantee that is a key that cannot collide, rather than a threshold you hope is tight enough.
What must never be cached:
- Anything containing another customer's data. The key must include the identity whose data appears in the answer, or the entry must not exist.
- Anything reflecting mutable account state — balance, current usage, order status. These look cacheable and produce confidently stale answers about the one thing the customer is actually asking about.
- Anything the answer to which changes when permissions change. Revoking access must not leave a cached answer reachable.
The measurement that decides the thresholds: collect pairs of real queries that are close in embedding space and check by hand whether they should share an answer. That set is what sets the similarity threshold; a number chosen without it is a guess with a data-exposure failure mode.
Your semantic cache reports a 40% hit rate and the team is pleased. What would you check before accepting that as a success?
Write your own answer first, on paper or in a note. Then compare.
Show a hint
A hit rate says how often it fired, not whether it was right to.
Show a strong answer
A hit rate is a usage metric, not a quality metric. Four things I would check:
- What fraction of hits were correct? Sample hits and compare the served answer against what a fresh generation would have produced. A 40% hit rate with 5% wrong answers is not a saving — it is a defect rate of 2% of all traffic, delivered confidently, and it will be discovered by a customer.
- What is the saving actually worth? 40% of calls avoided sounds large; if the cached ones are the short, cheap questions and the expensive long ones all miss, the cost saving may be a fraction of that. Measure saved tokens and saved latency, not saved calls.
- Is the hit rate concentrated? If a handful of queries account for most hits, a simple exact-match cache on normalised text would capture nearly the same benefit with no similarity risk at all. That is a strictly better design and the numbers may already justify it.
- What is the staleness distribution? How old is the average served answer, and has anything it depends on changed since? A hit against an entry built from a document deleted last week is a serious problem that the hit rate presents as a success.
The reframing I would offer the team: the metric to optimise is cost per correct answer, not hit rate. Hit rate can be raised at will by loosening the threshold, and every point gained that way is bought with correctness. Presenting the two numbers together is what keeps the optimisation honest.
Key takeaways
- In AI systems, caching is primarily a cost control, and it is usually the largest single lever available.
- Exact caching is trivially safe; include every parameter in the key and scope by user when prompts contain user data.
- Prefix caching reuses the prefill work for a shared prompt beginning — so put stable content first, variable content last.
- Semantic caching matches questions that mean the same thing, and the threshold is a correctness decision, because close-but-opposite questions exist.
- Never share semantic cache entries across users, and segment by tenant and language.
- Invalidate when source documents change, or you will quote superseded material confidently.
- Track hit rate, cost saved and wrong-hit rate together; the third is what keeps the first two honest.
Further reading
- RFC 9111, HTTP Caching — freshness and validation, which apply unchanged to answer caches.
- Fu Bang, "GPTCache: An Open-Source Semantic Cache for LLM Applications" (NLP-OSS 2023).
- Rajesh Nishtala et al., "Scaling Memcache at Facebook" (NSDI 2013) — stampedes and invalidation at scale.
- Nelson Liu et al., "Lost in the Middle" (TACL, 2024) — prompt ordering, which determines what a prefix cache can reuse.
- Berk Atikoglu et al., "Workload Analysis of a Large-Scale Key-Value Store" (SIGMETRICS 2012) — measured skew, which is what makes exact-match caching effective.