Backpressure and Rate Limiting: The Two Answers Candidates Skip

There's a moment in most system design interviews where the candidate says "we'll put a queue in front of it to handle traffic spikes." It's usually correct and it's usually the end of their thinking, which is a shame, because the interesting question starts one step later: what happens when the spike doesn't stop?

A queue doesn't absorb load. It defers it. If arrivals exceed service rate for long enough, an unbounded queue converts a fast failure into a slow one — memory climbs, latency climbs with queue depth, and every request that eventually gets served has been waiting so long the client gave up. You've traded 503s for timeouts, and timeouts are worse, because the client already retried.

The two mechanisms that actually deal with this are backpressure and rate limiting. They're often used interchangeably in interviews. They're not the same thing.

Backpressure: the receiver says stop

Backpressure is a downstream signal travelling upstream. A component that's saturated refuses new work rather than accepting it into an unbounded buffer, and that refusal propagates back toward the source.

The important word is refuses. A bounded queue that drops when full is backpressure. An unbounded queue is the absence of backpressure, regardless of what's in front of it.

What it protects against is the cascade. Without it, an overloaded database doesn't fail alone — it holds connections open, the application servers block waiting on those connections, their own request queues fill, the load balancer's health checks start timing out, and now the whole tier is down because of one slow component. The failure spread because every layer was willing to accept work it couldn't complete.

With backpressure the database rejects immediately at capacity, the app server gets a fast error and can shed that request, and the blast radius stays where the problem started. You still fail — but you fail at the boundary, quickly, with a clear signal, and you recover the moment load drops.

The follow-up to be ready for: what should the caller do with the rejection? "Retry" is the wrong instinct — naive retries under load are how you turn a brownout into an outage. The answer involves exponential backoff with jitter, a retry budget, and a circuit breaker that stops trying entirely once the failure rate crosses a threshold.

Rate limiting: the sender is capped

Rate limiting is a policy applied at the edge, before work enters the system at all. It's not a response to saturation — it applies whether you're busy or idle. It says this client gets N requests per second, full stop.

The canonical implementation is the token bucket: tokens refill at a fixed rate up to a maximum, each request consumes one, and requests arriving at an empty bucket are rejected. The bucket size is your burst allowance; the refill rate is your sustained rate. That's why token bucket is the standard answer — it permits short bursts (real traffic is bursty) while bounding the long-run average, which a simple fixed-window counter can't do without the boundary-spike problem.

What rate limiting protects against is different from backpressure: it's fairness and abuse, not saturation. One client's runaway retry loop shouldn't consume the capacity that every other client paid for. Backpressure has no opinion about who the load came from; rate limiting is entirely about that.

The distinction, stated plainly

Backpressure is reactive, internal, and about survival. It triggers on saturation and its job is to stop a local failure from becoming a global one.

Rate limiting is proactive, edge-facing, and about policy. It triggers on a configured threshold and its job is to bound what any one caller can consume.

You need both, and they fail in different directions. A system with rate limiting but no backpressure survives abusive clients and dies from an internal slowdown — a degraded database will still take the whole system with it, because nothing internal ever says stop. A system with backpressure but no rate limiting stays up under internal failure but lets one client's bad retry loop crowd everyone else out.

Watch the queue depth

The cheapest way to internalise this is to watch a queue's depth over time under sustained overload, rather than reasoning about it abstractly.

In ArchiDepth, nodes have real capacity and reject when saturated, and gateway components implement token-bucket limiting with a configurable rate. Build a source, a gateway and a backend, set the backend's capacity below the arrival rate, and run it. The first thing you notice is that queue depth doesn't plateau — it climbs monotonically, and the requests coming out the far end are older and older. Then set a rate limit at the gateway and run the same traffic. Throughput drops, some requests get dropped at the edge, and the requests that do get served come back fast.

Losing some requests quickly beats serving all of them too late. That's the whole lesson, and it's much more convincing on a chart than in a paragraph.