Recall two numbers from the fundamentals: reading from RAM takes about 100 nanoseconds, reading from disk about 10 milliseconds. Memory is roughly 100,000 times faster.
Caching is the entire discipline built on that gap. Keep the answers you need most in the fast place, and only pay the slow cost when you must.
It is the highest-leverage optimisation in system design — and the one most likely to introduce subtle bugs, because a cache is by definition a second copy of the truth.
Why it works: skew
Caching would be pointless if access were uniform. It is not. Real traffic is enormously skewed — a small fraction of items receive most of the requests.
One celebrity's profile may be read a million times a day while most users' profiles are read twice. Caching the top 1% of items can serve 90% or more of requests. That asymmetry is what makes a small, expensive memory store worth having in front of a large, cheap disk store.
Before adding a cache, ask whether your access pattern is actually skewed. If every item is equally likely, a cache mostly adds complexity and misses.
Skew is what buys the hit rate, and the hit rate is what buys the latency. Move the two sliders below to see how much — and notice how steep the last few percent are.
The curve is why teams chase hit rate so hard, and it is also the trap: the average falls fast while the misses stay exactly as slow as they always were. A 95% hit rate means one request in twenty still pays full price, and those requests are what your p99 is made of.
Where caches live
There is not one cache. There is a chain, and each layer catches what the layer above missed.
| Layer | Holds | Typical latency |
|---|---|---|
| Browser | Assets, API responses | 0ms |
| CDN | Static content, public responses | 10–50ms |
| Load balancer / proxy | Whole responses | ~1ms |
| Application (Redis/Memcached) | Query results, sessions, computed data | ~1ms |
| Database | Query and page cache | Internal |
This lesson focuses on the application layer — a shared in-memory store such as Redis sitting between your servers and your database.
Reading: cache-aside
The dominant pattern, and the default you should assume unless there is a reason otherwise.
async function getUser(id) {
const cached = await redis.get(`user:${id}`);
if (cached) return JSON.parse(cached); // hit
const user = await db.users.findById(id); // miss
await redis.set(`user:${id}`, JSON.stringify(user), 'EX', 300);
return user;
}The application owns the logic: look in the cache, fall back to the database, populate on the way out. It is simple, and it survives cache failure — if Redis is down every request just goes to the database, slower but working.
The alternative, read-through, hides this inside a library or the cache itself, so your code just asks the cache and it fetches on a miss. Cleaner code, less control.
Writing: three strategies with real trade-offs
Reads are easy. Writes are where correctness is decided, because you now have two copies of the data.
Write-through — write to cache and database together, synchronously. The cache is always consistent with the database. Writes are slower, and you cache things that may never be read.
Write-behind — write to the cache and flush to the database asynchronously. Very fast writes, and it absorbs bursts. But if the cache dies before flushing, you lose data. Only acceptable when loss is tolerable — view counts, analytics.
Write-around — write straight to the database and simply invalidate the cache entry. The next read repopulates it. This is the pragmatic default for most applications, because it avoids caching write-heavy data nobody reads.
Eviction: what leaves when memory is full
A cache is deliberately smaller than the data it fronts, so something must go.
- LRU (least recently used) — evict what has not been touched longest. The sensible default for almost everything.
- LFU (least frequently used) — evict what is used least often. Better when popularity is stable, but a formerly popular item can linger.
- TTL — everything expires after a set time. Not really an eviction policy but a freshness control, and usually combined with the others.
TTL deserves a moment of thought. It is your answer to "how stale may this be?" A product price might tolerate 60 seconds. A user's permissions probably should not. Choosing a TTL is choosing how wrong you are willing to be.
The three failure modes
These are what interviewers probe, and what actually causes outages.
Cache stampede
A popular key expires. A thousand concurrent requests miss simultaneously and all hit the database at once, which falls over.
Fixes: a lock so only one request rebuilds while others wait; serving stale data during the refresh; jittering TTLs so keys do not expire together.
Cache penetration
Requests for data that does not exist. Every one misses, hits the database, finds nothing, and caches nothing — so the cache provides no protection at all. An attacker requesting random ids exploits this deliberately.
Fixes: cache the negative result too, with a short TTL; or use a Bloom filter to reject ids that definitely do not exist.
Cache avalanche
A large number of keys expire at the same moment — or the cache restarts empty — and the full traffic load lands on the database at once.
Fixes: randomised TTLs; warming the cache before taking traffic; rate limiting the origin.
Diagram, as text
- Popular key expires leads to 1000 concurrent misses
- 1000 concurrent misses leads to Database
- Database leads to Overload
- Fix: one rebuilds, others wait or serve stale leads to 1000 concurrent misses
Consistency: the price you agreed to pay
The moment you cache, you accept that the cached copy can disagree with the database. That is not a bug to eliminate; it is the trade you made for speed.
What matters is deciding per piece of data how much staleness is acceptable — the same reasoning as the CAP lesson, applied at a smaller scale.
- A follower count 30 seconds stale: fine.
- A product price 30 seconds stale: probably fine, and a real risk at checkout.
- A user's permissions 30 seconds stale after being revoked: a security problem.
For the last category, invalidate explicitly on change rather than relying on TTL.
Check your understanding
Common mistakes
Caching before measuring. A cache is complexity and a second source of truth. Confirm the query is actually slow and actually repeated first.
No TTL at all. Data that never expires drifts from reality forever and fills memory.
Caching the wrong thing. Caching a query that runs once a day helps nobody. Cache what is hot.
Ignoring cache size. When memory fills, eviction begins and hit rate collapses — often the real cause of a sudden latency increase.
Making the cache required. If your system fails when Redis is down, the cache stopped being an optimisation and became a dependency. Degrade to the database.
Not monitoring hit rate. It is the one metric that says whether the cache is doing anything. A falling hit rate is an early warning.
Practice
A product page shows a price, a stock count and a review score. Choose a caching approach and TTL for each field, and justify each choice by what a stale value actually costs.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Ask, for each: if this is 60 seconds out of date, who is harmed and how much?
Show a strong answer
- Review score — cache aggressively, TTL of an hour or more. A score computed from thousands of reviews barely moves, and a stale one harms nobody. This is the easy case and the one where caching is pure profit.
- Price — cache with a short TTL, and invalidate explicitly on change. A stale price is not merely untidy: showing £10 and charging £12 is a broken promise, and in some jurisdictions a legal one. So cache it for speed, but delete the key the moment a price changes rather than waiting for expiry, and re-validate the price at checkout against the source of truth so the cached value is never what the customer is actually charged.
- Stock count — cache very briefly, or not at all, and never trust it for the decision. Displaying "3 left" when there are 2 is harmless. Allowing a purchase because the cache said 3 is an oversell. The pattern is to cache the display value and check the real one on the write path — which is the same shape as the price answer, and the general rule for anything scarce.
The organising principle: cache what is read, verify what is decided. A cache is for making reads cheap, not for making decisions correct.
Your cache hit rate is 94% and the team wants to raise it to 99% by increasing memory fourfold. Estimate whether that is worth doing, and what you would measure first.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Work out what fraction of total latency the misses actually account for before pricing the memory.
Show a strong answer
Do the arithmetic before buying anything.
Say a hit costs 1 ms and a miss costs 50 ms. At 94% hit rate, average latency is (0.94 × 1) + (0.06 × 50) = 3.9 ms. At 99% it is (0.99 × 1) + (0.01 × 50) = 1.5 ms. So the average improves by about 2.4 ms — a 60% reduction, which sounds impressive and may be worth very little if 3.9 ms was already far below anything a user notices.
What matters more is what the misses do to the thing behind the cache. Going from 6% to 1% of misses cuts database load by roughly a factor of six. If the database is the constrained resource — and it usually is — that is the real return, and it may be the difference between needing a larger database and not.
What I would measure first:
- Where the misses come from. If they are a long tail of keys read once, more memory buys almost nothing: those keys will still be cold next time. If they are evictions of keys that would have been hit again, more memory converts directly into hits. Eviction rate versus cold-start rate separates these.
- What a miss actually costs, in database load rather than in milliseconds. That is the number that justifies the spend.
- Whether the hot set has a natural size. If 99% of reads come from 20 GB of keys, the fourfold increase past that point is wasted; the curve flattens hard once the working set fits.
The honest conclusion is often that 94% is fine and the money belongs elsewhere — but the database-load argument is the one that occasionally makes it worth it.
Key takeaways
- Caching exploits the ~100,000x gap between memory and disk, and works because real access patterns are heavily skewed.
- Caches form a chain from browser to database; the application-level shared store is where most design decisions happen.
- Cache-aside is the default read pattern and degrades gracefully when the cache is unavailable.
- On writes, prefer write-around plus invalidation, and delete keys rather than updating them to avoid a permanent stale value from a race.
- TTL is a decision about how wrong you are willing to be. Invalidate explicitly for security-sensitive data.
- Know the three failure modes — stampede, penetration, avalanche — and their fixes.
- A cache must remain an optimisation, not a dependency.
Next: the thing the cache is protecting — the database itself, and the first real choice you make about it.
Further reading
- Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017), chapter 1 — caching in the context of load parameters.
- Berk Atikoglu et al., "Workload Analysis of a Large-Scale Key-Value Store" (SIGMETRICS 2012) — measured skew, hit rates and object sizes in a production cache.
- Rajesh Nishtala et al., "Scaling Memcache at Facebook" (NSDI 2013) — stampedes, leases and cache invalidation at scale.
- Pei Cao and Sandy Irani, "Cost-Aware WWW Proxy Caching Algorithms" (USENIX 1997) — why eviction policy depends on what a miss costs.
- Phil Karlton, attributed — "There are only two hard things in Computer Science: cache invalidation and naming things."