The low-level lesson on rate limiting covered the algorithms — fixed window, sliding window, token bucket, leaky bucket — and how to implement one correctly.
This lesson is the architectural half: where the limiter goes in a distributed system, how thousands of nodes agree on a count, and how limiting fits into keeping a system alive under load. Those are different questions, and interviews at this level ask the second set.
Where to enforce
The single most consequential decision. The same algorithm behaves very differently depending on where it runs.
- The client's request passes through a CDN edge, then an API gateway.
- The gateway forwards to the service, which reads and writes its database.
Describe this diagram
Components
- client (Client)
- edge (Content delivery network)
- gateway (API gateway)
- service (Service)
- db (PostgreSQL)
Connections
- client sends to edge
- edge sends to gateway
- gateway sends to service
- service sends to db
At the edge (CDN). Rejected before entering your infrastructure. Cheapest possible rejection and the only layer that meaningfully absorbs volumetric attacks — by the time traffic reaches your gateway you are already paying for it. The limitation is that the edge knows little about your users; it works with IPs and API keys, not business context.
At the API gateway. The usual home for per-customer quotas. The gateway already authenticates, so it knows who is calling and can apply per-plan limits. One implementation protects every service behind it.
In the service. Necessary for limits that depend on business rules — "three password resets per account per day" needs to know about accounts. Also the last line of defence for internal callers that bypass the gateway.
At the database. Connection pool limits are a crude rate limiter, and worth naming: when the pool is exhausted, requests queue. That queue is the thing that actually takes systems down.
The practical answer is several layers, each with a different job: coarse volumetric limits at the edge, per-customer quotas at the gateway, business-rule limits in services.
The distributed counting problem
This is the heart of the topic, and where the LLD lesson's warning becomes an architecture question.
You run 50 gateway instances. A customer's limit is 1,000 requests per minute. Each instance sees roughly 1/50th of their traffic. With local counters, the effective limit is 50,000 — fifty times what you configured.
Three approaches, with real trade-offs.
Centralised counters
Every instance increments a shared store — typically Redis — atomically.
const key = `rl:${customerId}:${Math.floor(Date.now() / 60000)}`;
const count = await redis.incr(key);
if (count === 1) await redis.expire(key, 120);
if (count > limit) return reject();Accurate, and simple to reason about. The costs are a network round trip on every request, and a new dependency in the hot path — if Redis is unreachable, you must decide whether to fail open (allow everything, risking overload) or fail closed (reject everything, causing an outage from a cache failure). Fail open is usually correct for rate limiting, because a limiter is a safeguard, not a security boundary.
Local counters with a shared budget
Each instance takes a slice of the quota locally and periodically reconciles with the shared store.
Near-zero latency, and it degrades gracefully. The trade is approximation — you may allow somewhat more than the limit during a sync interval. For most quotas that is fine, and it is how large-scale limiters actually work.
Sticky routing
Route each customer consistently to the same instance so its local counter is authoritative. Uses consistent hashing, and inherits its properties: cheap, but rebalancing disturbs counts and a hot customer concentrates on one node.
| Approach | Accuracy | Latency | Failure behaviour |
|---|---|---|---|
| Centralised | Exact | Round trip per request | Dependent on the store |
| Local + sync | Approximate | None | Degrades gracefully |
| Sticky routing | Exact per customer | None | Disturbed by rebalancing |
Rate limiting as load shedding
Beyond blocking abuse, limiting is a reliability mechanism, and this framing is what separates the HLD view from the implementation view.
Recall from the fundamentals that as arrival rate approaches capacity, latency does not degrade gently — it goes near-vertical, because the queue grows faster than it drains. Then clients time out and retry, which adds load, which lengthens the queue. A system in that state does not recover on its own.
Rejecting 10% of traffic immediately is vastly better than serving 100% of it slowly, because the fast rejection keeps the queue short and the system in a regime it can recover from.
Two refinements worth naming:
Prioritised shedding. Not all traffic is equal. Under pressure, shed background jobs and analytics before user-facing reads, and shed anonymous traffic before paying customers.
Adaptive limits. Rather than a fixed number, derive the limit from observed health — current latency, queue depth, error rate. When P99 latency rises, tighten automatically. This is more robust than a static limit chosen a year ago against different hardware.
Multi-tenancy and fairness
When many customers share infrastructure, one customer's burst must not consume everyone's capacity — the noisy neighbour problem.
Per-customer quotas handle the obvious case. Two refinements matter at scale:
Fair queuing. Rather than first-come-first-served, serve customers in rotation so a single heavy caller cannot monopolise workers.
Global backstop. Alongside per-customer limits, cap total system throughput. Ten thousand customers each staying just under their individual limit can still collectively exceed what you can serve.
What to expose
Limits must be discoverable, or clients cannot cooperate:
HTTP/1.1 429 Too Many Requests
Retry-After: 30
X-RateLimit-Limit: 1000
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1718020800Send X-RateLimit-Remaining on successful responses too, so well-behaved clients slow down before hitting the wall rather than discovering it by being blocked.
Check your understanding
Common mistakes
Only limiting at the application layer. By then you have paid to accept, parse and authenticate the request.
Failing closed when the limiter's store is down. A Redis outage should not become a full outage. Allow traffic and alert.
No limits on internal traffic. A retry storm between your own services is a common cause of cascading failure.
Returning 500 instead of 429. Clients treat 5xx as transient and retry hard; 429 with Retry-After is an instruction.
One global limit for all endpoints. A cached read and a report generator have wildly different costs and deserve different limits.
Forgetting the limiter is a shared dependency. If every request consults one Redis instance, that instance is now a single point of failure for your entire API.
Practice
Design rate limiting for a public API with a free tier (100 requests/hour), a paid tier (10,000/hour), and internal services that must never be limited. Specify the algorithm, where it runs, and what the response says.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Decide what identity the limit is keyed on before choosing an algorithm.
Show a strong answer
Key on the API key, not the IP address. IPs are shared behind corporate NAT and mobile carriers, so IP limiting punishes unrelated users together; and a determined caller changes IP trivially. The API key identifies the account, which is what the tier belongs to.
Algorithm: token bucket. It permits a burst up to the bucket size and then settles to the refill rate, which matches how real clients behave — a page load fires ten requests at once and then nothing for a minute. A strict per-second limit would reject that legitimate burst; a fixed window would allow double at boundaries.
- Free: bucket of 100, refill 100/hour.
- Paid: bucket of 10,000, refill 10,000/hour, with a smaller per-second cap on top so one client cannot consume the whole hour's allowance in a second and saturate the backend.
- Internal: identified by a separate credential and exempt — but still counted and reported, because an internal caller in a retry loop is the most likely source of an incident, and an exemption you cannot observe is a blind spot.
Where it runs: at the gateway, before authentication does any expensive work, with shared counters so it holds across instances.
What the response says. Status 429, plus:
Retry-Afterwith a concrete number of seconds — without it, clients retry immediately and make it worse.- Headers giving the limit, remaining allowance and reset time, on every response rather than only on rejections, so a well-behaved client can slow down before being refused.
The distinction worth stating: the headers are what turn rate limiting from a punishment into a protocol.
During an incident your service is overloaded. Explain the difference between rate limiting and load shedding, and design what you would drop first.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
One is about fairness between clients; the other is about survival.
Show a strong answer
Rate limiting is a policy applied at all times: this client gets this much, because that is what they are entitled to. It is about fairness and abuse prevention, and it is configured in advance.
Load shedding is a reaction to the system's own state: we are over capacity now, so some work must be refused regardless of entitlement. A paying customer within their limit may still be shed, because the alternative is that everyone fails.
Conflating them is a common and expensive mistake — a system with rate limits but no shedding will happily accept every request that is within limit right up to the point it falls over.
What I would drop, in order:
- Requests already past their deadline. If a client's timeout is 1 second and a request has queued for 3, completing it helps nobody and consumes capacity. Check the remaining budget before starting work.
- Retries before first attempts. A retry means someone already got an answer, even if it was a failure. Prioritising first attempts prevents the retry storm that turns overload into collapse. This requires clients to mark retries, which is worth asking for.
- Expensive, low-value work. Analytics, recommendations, background recomputation — anything whose absence degrades rather than breaks.
- Lowest tier first, but never to zero: shedding 100% of free traffic makes the product look completely down to the largest group of users. Shed proportionally.
Never shed: health checks, and anything on the path to recovery. Shedding your own health endpoint under load gets every instance removed from rotation at the moment you most need them.
Key takeaways
- Where you enforce matters as much as the algorithm: the edge sheds volume cheaply, the gateway knows identity and enforces plans, services enforce business rules.
- With N instances, local counters multiply your limit by N. Fix with centralised atomic counters, local counters plus reconciliation, or sticky routing.
- Fail open when the limiter's store is unavailable — it is a safeguard, not a security boundary.
- Rate limiting is load shedding: rejecting 10% fast beats serving 100% slowly, because it keeps the queue short enough to recover.
- Prefer prioritised and adaptive shedding over a fixed number chosen long ago.
- Always return 429 with
Retry-After, and expect clients to need backoff and jitter or they will amplify the incident. - Add a global backstop alongside per-customer quotas, since many compliant tenants can still exceed total capacity.
Further reading
- RFC 6585, Additional HTTP Status Codes — the definition of 429 Too Many Requests.
- RFC 9239 draft series, RateLimit header fields for HTTP — the emerging standard for advertising limits.
- Google SRE, Site Reliability Engineering (O'Reilly, 2016), chapter 21 — handling overload, criticality and adaptive throttling.
- Jeffrey Mogul and Ramana Kompella, "Inferring the Network Latency Requirements of Cloud Tenants" (HotOS 2015) — deadlines as a shedding signal.
- Michael Nygard, Release It!, second edition (Pragmatic Bookshelf, 2018) — the difference between back pressure and shedding.