"The system is slow." That sentence is useless to an engineer, because it can mean two completely different things with completely different fixes.
It might mean one request takes a long time. Or it might mean the system cannot handle enough requests at once. These are separate problems, and confusing them leads teams to spend months optimising the wrong thing.
The highway
Picture a motorway between two cities.
- Latency is how long one car takes to make the journey. Measured in time: milliseconds, seconds.
- Throughput is how many cars pass a given point per hour. Measured as a rate: requests per second.
Now the crucial insight — these are independent. You can change one without changing the other.
Add three more lanes. Throughput triples: far more cars get through per hour. But any individual car still drives the same distance at the same speed, so latency is unchanged. The journey takes exactly as long; there is simply more room.
Raise the speed limit instead. Now each car arrives sooner — latency drops — and as a side effect more cars pass per hour too.
Diagram, as text
- High throughput leads to High latency
- Low throughput leads to Low latency
Why this distinction pays off
Suppose users complain the app is slow. You measure and find requests take 2 seconds. Where does that time go?
- If one user alone on the system still waits 2 seconds, you have a genuine latency problem. Something in the request path is slow: an unindexed query, a call to a sluggish third party, too many sequential round trips.
- If a single user gets a reply in 50ms but it degrades to 2 seconds under load, you have a throughput problem. Each request is fine; there are simply more arriving than you can serve, so they queue.
The fixes have nothing in common. The first needs profiling and optimising the slow step. The second needs capacity — more servers, a cache, a bigger connection pool. Applying the wrong fix wastes a quarter.
Why averages lie
This is the most important practical lesson in the topic.
Suppose you measure 100 requests. Ninety-nine take 10ms. One takes 5 seconds. The average is about 60ms, which looks excellent — you would report that number happily.
But a real person waited five seconds. If you serve a million requests a day with that profile, ten thousand people had a terrible experience while your dashboard said everything was fine.
Averages hide the worst cases, and the worst cases are what users remember. So engineers use percentiles.
| Metric | Meaning | Reading it aloud |
|---|---|---|
| P50 (median) | Half of requests are faster than this | "The typical experience" |
| P95 | 95% are faster; 1 in 20 is worse | "A bad day for a few users" |
| P99 | 99% are faster; 1 in 100 is worse | "The tail — the complaints" |
| P99.9 | 999 in 1000 are faster | "The pathological cases" |
A P99 of 2 seconds means one request in every hundred takes longer than two seconds. At a million requests a day, that is ten thousand slow experiences daily.
What actually causes latency
Latency is mostly unavoidable physical cost plus avoidable mistakes. Knowing the rough scale of each helps enormously.
| Operation | Rough time | Relative to RAM |
|---|---|---|
| Read from RAM | ~100 nanoseconds | 1x |
| Read from SSD | ~100 microseconds | ~1,000x slower |
| Read from spinning disk | ~10 milliseconds | ~100,000x slower |
| Network round trip, same datacentre | ~500 microseconds | ~5,000x slower |
| Network round trip, across the world | ~150 milliseconds | ~1,500,000x slower |
Two conclusions fall straight out of this table.
Memory is dramatically faster than everything else. This is the entire justification for caching — an answer already in RAM avoids a trip costing a thousand times more.
Distance is expensive and cannot be optimised away. A signal cannot travel faster than light. London to Sydney is roughly 17,000km, so a round trip has a floor near 110ms no matter how good your code is. This is why content delivery networks exist: the only way to beat the speed of light is to not make the trip, by keeping a copy near the user.
What actually limits throughput
Throughput is capped by whichever resource saturates first — the bottleneck. Usually one of:
- CPU: the server is computing as fast as it can.
- Memory: no room for more concurrent work.
- Disk or network I/O: data cannot move fast enough.
- Connection or thread limits: often a database allowing only N connections. This is the most common surprise, because the machine looks idle while requests queue for a connection slot.
Adding capacity anywhere except the bottleneck changes nothing. Ten application servers waiting on one database's connection pool have the throughput of one.
The relationship under load
The two are independent until the system approaches its limit. Then they couple violently.
As the arrival rate nears capacity, a queue forms. Every request now waits behind others, so latency rises. Push a little further and latency does not rise gently — it goes near-vertical, because the queue grows faster than it drains.
This is why a system can look healthy at 70% utilisation and fall over at 85%. The warning sign is not throughput, which stays flat at maximum, but latency climbing while throughput does not. That is the signature of a queue building, and it is the most useful early alarm you can watch.
Drag the slider and watch where the curve turns. The work each request does never changes — only the waiting does.
Notice what the shape means for capacity planning. The distance from 50% to 70% utilisation costs you very little. The distance from 85% to 95% costs you everything. Headroom is not waste; it is the only thing standing between you and the vertical part of that curve.
Check your understanding
Common mistakes
Reporting averages. An average can look excellent while thousands of users suffer. Report P50 and P99 together — the pair tells the real story.
Optimising code when the problem is queueing. If a lone request is already fast, your functions are not the problem; capacity is.
Ignoring the network. Beginners tune algorithms while the real cost is five sequential API calls, each a full round trip. Reducing round trips usually beats micro-optimising code.
Measuring on a fast laptop next to the server. Your users are on mobile networks, possibly on another continent. Measure where they are.
Practice
A page makes five sequential API calls, each taking 40 ms of server time, to a server 100 ms away. Work out what the user waits, then show what changes if you parallelise the calls, and what changes if you combine them into one endpoint.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Separate the fixed cost of the trip from the cost of the work.
Show a strong answer
Sequential: each call is one round trip (100 ms) plus server time (40 ms) = 140 ms. Five of them = 700 ms.
Parallel: all five leave together, so you pay one round trip and the slowest server time: 100 + 40 = 140 ms. The 560 ms saved was pure waiting.
Combined into one endpoint: one round trip, and the server does all five pieces of work — in parallel internally, so ~40 ms, or 200 ms if it does them in series. Best case 140 ms, and now only one connection, one set of headers and one authentication check.
The lesson: 560 of the original 700 ms was not work, it was trips. Parallelising is usually the cheaper change; combining endpoints wins additionally on overhead and is worth it when the client always needs all five together.
Your team must choose between two changes: one halves P50 from 200 ms to 100 ms, the other halves P99 from 8 s to 4 s. The service takes 50 million requests a day. Which would you pick, and what would change your mind?
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Count the people affected by each, then ask what each group does next.
Show a strong answer
Count first. The P99 change affects the slowest 1% — 500,000 requests a day. The P50 change affects roughly half — 25 million. On volume alone the P50 change looks fifty times bigger.
Then ask what the users do. Saving 100 ms on an already-fast request is usually imperceptible; a user who waited 8 seconds may have abandoned the page, retried (adding load), or filed a complaint. Tail latency is where sessions are lost, and retries from timed-out requests make the tail self-reinforcing.
Take P99 first, in most cases — the 500,000 are the ones actually suffering, and tail failures generate the retries that degrade everyone else.
What would change my mind:
- If the P50 path is on a critical interaction repeated many times per session (a search-as-you-type box), 100 ms × many is a real experience change.
- If the P99 tail is dominated by a small set of pathological inputs — one customer with a million-row report — then the fix is that query, not a system-wide investment.
- If the tail is caused by clients that have already given up, improving it changes nothing anyone sees.
Key takeaways
- Latency is time per request; throughput is requests per second. Separate dials, completely different fixes.
- Diagnose by comparing idle latency with peak latency: bad when idle means a slow path, bad only at peak means a queue.
- Averages hide the worst cases. Use P50 alongside P99, because the tail is what users complain about.
- Memory is ~1,000x faster than disk, and distance costs milliseconds you cannot optimise away — the justifications for caching and CDNs respectively.
- Throughput is set by the bottleneck; adding capacity elsewhere achieves nothing.
- Near capacity the two couple: latency spiking while throughput stays flat means a queue is forming, and it is your best early warning.
Next we look at what happens when parts of the system stop responding altogether, and the difference between being up and being correct.
Further reading
- Jeffrey Dean and Luiz André Barroso, "The Tail at Scale" (Communications of the ACM, 2013) — why P99 dominates user experience in fan-out systems.
- Gil Tene, "How NOT to Measure Latency" (Strange Loop, 2015) — coordinated omission, and why most latency numbers are wrong.
- Leonard Kleinrock, Queueing Systems, Volume 1: Theory (Wiley, 1975) — the queueing results behind the latency-under-load curve.
- John D. C. Little, "A Proof for the Queuing Formula L = λW" (Operations Research, 1961) — the relationship between throughput, latency and concurrency.
- Brendan Gregg, Systems Performance (Pearson, 2020), chapter 2 — the USE method for locating a saturated resource.