You have three application servers. A request arrives. Which one handles it?
Something must decide, and that something is a load balancer. It sounds like a small job — pick a server — but the decision quietly determines whether your system stays up when a machine dies, whether users get logged out at random, and whether adding capacity actually helps.
What it really buys you
Spreading load is the obvious benefit. Two others matter more.
Fault tolerance. The load balancer continuously checks which servers are alive and stops sending traffic to ones that are not. A machine can die at 3am and users notice nothing. Without this, a third of your requests would hit a dead server.
Deployments without downtime. Take one server out of rotation, upgrade it, put it back, repeat. Users are served throughout by the others.
That second point is why load balancers appear even in systems that do not need the capacity.
The algorithms
Round robin
Requests go to each server in turn: 1, 2, 3, 1, 2, 3. Simple, needs no state, and distributes evenly by count.
Its weakness is that it counts requests, not work. If one request triggers a 5-second report and the next is a cached lookup, "even by count" is wildly uneven by load.
Least connections
Send each request to the server currently handling the fewest.
This adapts automatically when requests vary in cost — a server stuck on slow work accumulates open connections and stops receiving new ones. For most real applications this is the better default, and it is what you should reach for when request durations are unpredictable.
Weighted
Give each server a share proportional to its capacity. Essential when your fleet is mixed — a 16-core machine should take more than a 4-core one. Also useful for canary deployments: send 5% of traffic to the new version.
IP hash
Hash the client's IP and always map that client to the same server. This gives you session affinity, or "sticky sessions".
It solves the stateful-server problem from the scalability lesson — but as noted there, it is a trap. When that server restarts, every user pinned to it loses their session at once, and a server holding many long-lived users cannot shed load. Prefer moving session state to a shared store.
| Algorithm | Good for | Weakness |
|---|---|---|
| Round robin | Uniform, cheap requests | Ignores actual load |
| Least connections | Variable request cost | Needs connection tracking |
| Weighted | Mixed hardware, canaries | Weights need maintaining |
| IP hash | Forced affinity | Uneven, and restarts drop sessions |
Health checks: the part that actually matters
An algorithm is worthless if it keeps routing to a broken server. Health checks are what make a load balancer a reliability tool rather than a distributor.
Passive checks watch real traffic and remove a server after repeated failures. Free, but it takes user-facing errors to notice.
Active checks poll an endpoint such as /health every few seconds. This catches problems before users do, which is why it is standard.
The quality of that endpoint decides everything:
// Nearly useless — proves the process is running, nothing more
app.get('/health', (req, res) => res.send('ok'));
// Meaningful — proves it can actually serve
app.get('/health', async (req, res) => {
try {
await db.query('SELECT 1');
await cache.ping();
res.status(200).json({ status: 'ok' });
} catch (err) {
res.status(503).json({ status: 'degraded' });
}
});Layer 4 versus Layer 7
Load balancers operate at one of two levels, and the difference is what they can see.
Layer 4 works with IP addresses and ports. It forwards packets without inspecting them. Extremely fast, protocol-agnostic, and cheap — but it cannot make decisions based on content, because it cannot read any.
Layer 7 understands HTTP. It sees the URL, headers and cookies, which unlocks decisions L4 cannot make:
/api/* -> API servers
/images/* -> media servers
/ -> web serversIt can also terminate TLS, rewrite headers, compress responses, and route by cookie for A/B tests.
Choose L7 for HTTP applications — the flexibility is worth the small overhead, and modern L7 balancers are very fast. Choose L4 for raw throughput or non-HTTP protocols such as database connections and game traffic.
The load balancer is a single point of failure
Here is the question interviewers ask and beginners miss.
You added a load balancer so no single server outage kills the system. But now every request flows through one box. If it dies, all three healthy servers become unreachable.
Diagram, as text
- Users leads to Virtual IP
- Virtual IP leads to Load Balancer 1 active
- Virtual IP leads to Load Balancer 2 standby (failover)
- Load Balancer 1 active leads to Server 1
- Load Balancer 1 active leads to Server 2
- Load Balancer 1 active leads to Server 3
The standard answer is a pair sharing a virtual IP address. If the active one stops responding, the standby claims the IP and takes over in seconds. Cloud load balancers do this for you, which is a real part of their value — the managed service is already redundant across availability zones.
Where load balancers sit
They are rarely used once. A typical system has several tiers:
- Global, using DNS or anycast, directing users to the nearest region.
- Regional, spreading traffic across web servers in that region.
- Internal, between microservices, so the order service balances across payment service instances.
Each layer solves the same problem at a different scale.
Check your understanding
Common mistakes
A health check that only proves the process is alive. A server with a dead database connection pool passes return 'ok' and keeps receiving traffic it cannot serve.
Sticky sessions as an architecture. They work until a deploy, then log everyone out. Externalise session state instead.
Forgetting to drain connections. Removing a server instantly kills in-flight requests. Stop sending new requests, wait for existing ones to finish, then remove it.
No timeout on upstream requests. A hung server holds connections open until the balancer's own limits are hit, and the failure spreads.
Balancing to a single database. Ten servers behind a load balancer all queueing on one database gives you the throughput of one database. Find the real bottleneck first.
Practice
Design the health checking for a service where a request takes 30 seconds to complete and each server handles ten concurrently. State what the check tests, how often it runs, and the thresholds — then explain what each choice prevents.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Consider what happens to in-flight work when a server is removed, and how long a false positive costs you.
Show a strong answer
What the check tests. Two separate endpoints, because they answer different questions:
- Liveness — is the process itself functioning? Cheap, no dependencies. Failing this means restart the process.
- Readiness — can it accept new work? This checks the critical dependencies and the server's own capacity: if all ten slots are busy, report not-ready so the balancer stops sending more.
How often, and thresholds. With 30-second requests, reacting in one second is pointless — you cannot drain faster than the work takes. Check every 5 seconds; require 3 consecutive failures before removing (15 seconds) and 2 consecutive successes before returning.
What each choice prevents:
- Requiring consecutive failures prevents a single dropped packet or GC pause from evicting a healthy server.
- Requiring consecutive successes before returning prevents flapping, where a struggling server is repeatedly given traffic, fails, and is removed again.
- Separating readiness from liveness prevents a busy server from being restarted when all it needed was to be left alone for a moment.
- Connection draining is essential here: on removal, the balancer must stop sending new requests but let the in-flight ones finish, and the drain timeout must exceed 30 seconds or you kill work that was about to succeed.
Traffic is spread evenly across eight servers by round robin, but two of them sit at 95% CPU while the rest idle at 20%. Give three possible causes and how you would distinguish them.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Round robin balances requests, not work.
Show a strong answer
Round robin equalises the number of requests, which only equalises load if requests cost the same. Three causes:
- Uneven request cost. One customer runs enormous reports; whichever servers receive those requests do far more work. How to tell: compare request duration distributions per server, not request counts. If the counts match and the durations differ, this is it. Fix with least-connections or least-time balancing rather than round robin.
- Sticky sessions. If sessions pin clients to a server, a few heavy users concentrate on a few servers and stay there for hours. How to tell: look at whether the hot servers stay the same over time. Round robin alone moves the hot spot around; stickiness makes it persistent.
- The servers are not identical. Different instance types, a noisy neighbour on shared hardware, or two servers also running a background job. How to tell: check whether the hot pair is hot even at low traffic, and whether they are the same two machines after a redeploy that reshuffles placement.
The diagnostic that separates all three: hold request count constant and compare work done. If equal counts produce unequal CPU, the problem is not the algorithm's fairness — it is that fairness in requests was never what you wanted.
Key takeaways
- A load balancer's biggest benefits are fault tolerance and zero-downtime deploys, not just spreading traffic.
- Least connections is the better default when request costs vary; round robin only equalises counts.
- IP hash gives sticky sessions, which is usually a symptom of stateful servers rather than a solution.
- Health checks decide whether it is a reliability tool. A check that only proves the process runs is close to useless — but avoid checks that remove the whole fleet when a shared dependency wobbles.
- L7 sees HTTP and can route by path, cookie and header; L4 is faster and protocol-agnostic.
- The load balancer is itself a single point of failure — run an active-passive pair on a virtual IP, or use a managed one.
- Drain connections before removing a server, or you kill in-flight requests.
Next: a load balancer is one kind of intermediary sitting in front of your servers. There is a broader family, and the distinction between them is a classic interview question.
Further reading
- RFC 9110, HTTP Semantics, sections on intermediaries — what a proxy or balancer may and may not change.
- Google SRE, Site Reliability Engineering (O'Reilly, 2016), chapters 19–20 — load balancing at the front end and inside the datacentre.
- Michael Mitzenmacher, "The Power of Two Choices in Randomized Load Balancing" (IEEE TPDS, 2001) — why sampling two servers is nearly as good as checking all of them.
- Michael Nygard, Release It!, second edition (Pragmatic Bookshelf, 2018) — health checks, circuit breakers and failing open.
- Willy Tarreau, "Making Applications Scalable with Load Balancing" (HAProxy technical paper, 2006) — layer 4 versus layer 7 in practice.