Five Load Balancing Algorithms, and When Each One Betrays You
Round-robin, least-connections, IP hash, weighted and random: what each does to your tail latency, and when the popular default is the wrong choice.
5 min read
On this page
"Add a load balancer" is the easiest sentence in a system design interview. The follow-up — "which algorithm, and why?" — is where it gets interesting, because every one of these has a specific failure condition, and the default is frequently the wrong pick.
Round-robin
Each request goes to the next backend in sequence. Simple, stateless, perfectly even request distribution.
The word doing the work there is request. Round-robin balances request counts, not load. That's the same thing only when every request costs roughly the same and every backend is roughly identical.
Where it betrays you: heterogeneous request costs. If 5% of your requests are expensive report generations and 95% are cheap key lookups, round-robin will happily hand three consecutive expensive requests to the same backend while its neighbours idle. The distribution is even; the load is not. You'll see it as tail latency that's much worse than your average, with no single backend obviously "hot" in aggregate.
It also degrades badly with long-lived connections. Round-robin balances at connection time, so with WebSockets or gRPC streams a backend that got unlucky during a connection surge stays overloaded for hours.
Least connections
Route to whichever backend currently has the fewest in-flight requests. This is round-robin's answer to heterogeneous cost: a backend stuck on slow work naturally stops receiving new requests, because its in-flight count stays high.
For most request/response HTTP services, this is the better default and it's underused.
Where it betrays you: it can't distinguish "busy" from "broken". A backend that fails fast — instantly returning 500s — has an in-flight count near zero, which makes it look like the least loaded node in the pool. Least-connections will then preferentially route traffic to your one broken server. This is a real and fairly common outage shape, and the fix is that least-connections needs health checking to be safe, not optional.
IP hash
Hash the client's address and map it to a backend, so a given client consistently lands on the same one. This buys you session affinity without shared session storage.
Where it betrays you: two ways, both nasty.
First, distribution quality depends entirely on how your clients are distributed. A large fraction of traffic behind one corporate NAT or one mobile carrier gateway hashes to a single backend. You get a hotspot that no amount of scaling out fixes, because scaling out doesn't change the hash target for that client.
Second, the backend set changing reshuffles the mapping. Add or lose a node with naive modulo hashing and most clients get remapped at once — every affinity-dependent session invalidates simultaneously. This is exactly the problem consistent hashing exists to solve, and it's worth knowing that distinction cold, because "how would you make that rehash cheaper?" is a standard follow-up.
Weighted
Assign each backend a share proportional to its capacity. The honest algorithm for a heterogeneous fleet: your 16-core boxes take more than your 4-core boxes.
Where it betrays you: the weights are static and your capacity isn't. Weights are usually set once at provisioning and then never revisited, so they slowly stop describing reality — a noisy neighbour on shared hardware, a backend running a different build, a node with a degraded disk. Weighted routing keeps confidently sending a struggling machine its full configured share. Weighted is a good pick when capacity genuinely differs and a bad one when capacity varies over time, which is what least-connections handles for free.
Random
Pick a backend uniformly at random. It sounds unserious and it's better than people expect: with enough requests the distribution converges, and there's no state, no coordination and no synchronised behaviour across load balancers.
That last property matters more than it sounds. Multiple round-robin balancers can fall into lockstep and hammer the same backend simultaneously. Random can't.
Where it betrays you: small request volumes. Convergence is a large-numbers property. At low traffic you get visible imbalance, which makes random a poor fit for the exact low-volume, high-cost workloads where imbalance hurts most.
The variant worth naming in an interview is power of two choices: pick two backends at random, send to whichever has fewer connections. It gets you most of the benefit of least-connections at a fraction of the coordination cost, and it's what several production balancers actually do.
The short version
| Algorithm | Good default when | Breaks when |
|---|---|---|
| Round-robin | Uniform requests, uniform backends, short connections | Request costs vary; long-lived connections |
| Least connections | Request costs vary (most HTTP services) | A backend fails fast and looks idle |
| IP hash | You need affinity without shared session state | Clients cluster behind NAT; the backend set changes |
| Weighted | Backend capacity genuinely differs | Capacity varies over time; weights go stale |
| Random | High volume, many balancers, no coordination | Low request volume |
Try it rather than memorising it
Reading the table gets you through one follow-up question. Watching the behaviour gets you through all of them.
In the ArchiDepth canvas a load balancer's algorithm is a real setting, and the simulation actually routes by it. Put a few backends behind one, give them different processing latencies, and run the same traffic through each algorithm in turn. Round-robin's imbalance under uneven cost is visible in about ten seconds — the node queue depths diverge while the request counts stay identical. That image tends to stick better than the table does.