Every remote call is a bet that another machine will answer. Most of the time it does. The interesting engineering is entirely in the rest.
A dependency does not usually fail cleanly. It gets slow — and slow is worse than down, because a fast failure frees your resources while a slow one holds them. That distinction is the key to everything on this page.
Resilience patterns are the small set of behaviours that keep one struggling dependency from taking your service with it.
How one slow dependency takes down a healthy service
Your service has a pool of 200 workers. It calls a payment provider that normally answers in 50 ms, so each worker is free almost immediately.
The provider degrades to 30 seconds. Nothing has failed — requests still succeed eventually. But each worker is now held 600 times longer, the pool fills, and requests that have nothing to do with payments begin to queue. Within a minute your entire service is unresponsive.
This is resource exhaustion through a slow dependency, and it is the most common way a distributed system fails. Notice that nothing crashed. The dependency did not go down and neither did you — you simply ran out of workers.
Every pattern below exists to break some link in that chain.
Timeouts
The first and most important control. A call without a timeout is a call that can hang forever, and defaults are rarely what you want — many clients ship with no timeout at all.
Set two, because they answer different questions:
- Connection timeout — how long to wait to establish a connection. Short: a second or two. Failure here means the host is unreachable, and waiting will not help.
- Request timeout — how long to wait for a response once connected. Derived from the dependency's actual latency, not a round number.
Pick the value from measured behaviour: a little above the p99 of the dependency's normal response time. Set it at the p50 and you will abandon healthy requests; set it at 30 seconds and you have not really set one.
Timeouts must also shrink as you go deeper. If your caller gives you 3 seconds, calling a dependency with a 5-second timeout is pointless — the caller has already given up. This is a deadline, and propagating it down the call chain is what stops work continuing on a request nobody is waiting for any more.
Retries, and how they cause outages
A retry converts a transient failure into a success, which is why it is tempting. It also multiplies load at precisely the moment a system is least able to take it.
Three rules make retries safe.
Only retry what is safe to repeat. A timeout means you do not know whether the operation happened. Retrying a payment on a timeout can charge twice — unless the operation is idempotent, which is why idempotency keys and retries are inseparable.
Back off exponentially, with jitter. Retrying immediately hits a struggling service hardest. Doubling the wait each time gives it room. Jitter — a random component — is the part people omit and the part that matters most: without it, every client that failed at the same moment retries at the same moment, and the recovering service is knocked over by a synchronised wave. This is the thundering herd, and jitter is what breaks it up.
Cap the attempts, and cap them globally. Three attempts is usually plenty. Be careful of retries at several layers: three retries at each of three layers is twenty-seven calls, which is an accidental denial-of-service against your own dependency.
Circuit breakers
Diagram, as text
- Closed: calls pass through leads to Open: fail fast, no calls (failures exceed threshold)
- Open: fail fast, no calls leads to Half-open: allow a few trials (cool-off elapses)
- Half-open: allow a few trials leads to Closed: calls pass through (trials succeed)
- Half-open: allow a few trials leads to Open: fail fast, no calls (any trial fails)
Retrying a dependency that is thoroughly broken is waste. A circuit breaker notices sustained failure and stops calling for a while, failing immediately instead.
It has three states:
| State | Behaviour | Transition |
|---|---|---|
| Closed | Calls pass through; failures counted | Opens when the failure rate crosses a threshold |
| Open | Calls fail instantly, dependency untouched | Moves to half-open after a cooldown |
| Half-open | A few trial calls allowed | Closes if they succeed, reopens if not |
Two benefits, and the second is the one usually missed. It protects the caller, which fails fast and keeps its workers free rather than blocking on a doomed call. And it protects the dependency, which gets a period of reduced load in which to recover — a struggling service under continuous retry pressure often cannot.
Trip on a failure rate over a rolling window rather than a raw count, or a busy endpoint will trip on a normal background level of errors while a quiet one never trips at all.
Bulkheads
Named after ship compartments: a hull is divided so one flooded section does not sink the vessel.
Applied to a service, it means giving each dependency its own resource pool rather than sharing one. If payments, search and recommendations each have their own limited set of workers or connections, then a slow payment provider can exhaust only the payment pool. Search still works.
This directly prevents the cascade at the top of this page. It costs some efficiency — pools sized separately are less well utilised than one shared pool — and buys containment. That is usually the right trade for anything user-facing.
Graceful degradation
The last resort, and the one that decides what the user actually experiences.
When a dependency is unavailable, the choice is between failing the whole request and returning something less complete. A product page missing its recommendations is still a product page; a product page returning a 500 because the recommendation service is down is a lost sale.
Decide per dependency, in advance, whether it is critical (its failure fails the request) or optional (its failure degrades the response). Most teams have never made this list explicit, which means the answer is decided accidentally by whatever the code happens to do.
Serving slightly stale cached data during an outage is often the best available answer, and it is worth designing for deliberately rather than discovering under pressure.
Check your understanding
Common mistakes
No timeout at all. Many clients default to unbounded, and an unbounded call defeats every other protection.
One timeout value everywhere. It should come from each dependency's measured latency.
Timeouts that grow deeper in the call chain. Work continues on a request the caller already abandoned.
Retrying non-idempotent operations. A timeout does not tell you whether it happened.
Backoff without jitter. Synchronised clients knock the recovering service straight back over.
Retries stacked at multiple layers. Three at each of three layers is twenty-seven calls.
Circuit breakers tripping on counts rather than rates. Busy endpoints trip constantly; quiet ones never do.
One shared pool for every dependency. The slowest one gets to consume all of it.
Never deciding which dependencies are optional. The answer gets decided accidentally, during an incident.
Practice
Set timeouts for a request path where the client waits up to 3 seconds, the gateway calls a service, and the service calls a database. Give each timeout and explain why they must decrease inwards.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
A timeout deeper in the stack that exceeds the one above it can never fire usefully.
Show a strong answer
The rule is that each layer's timeout must be shorter than its caller's, or the inner timeout is unreachable — the caller has already given up, and the inner call is doing work for a response nobody will read.
A workable budget:
- Client: 3,000 ms. The user-facing promise.
- Gateway: 2,500 ms. Leaves ~500 ms for the gateway's own overhead, serialisation and the network hop back, so the client's deadline is met with margin.
- Service: 2,000 ms. Leaves the gateway room to return a useful error, or a partial response, rather than being cut off itself.
- Database query: 1,000 ms, with a connection-acquisition timeout of ~200 ms on top.
Why the database gets much less than the remainder: the service may need to make more than one call, and it needs time to do something sensible with a failure — fall back to a cache, return a degraded response, record a metric. A timeout budget that leaves zero room for handling the timeout is not a budget.
The better version of this is a deadline, not a set of timeouts. Pass the remaining time with the request: the client says "3,000 ms from now", each hop subtracts what it has used, and each hop refuses to start work it cannot finish in the time left. Fixed per-layer timeouts are an approximation of that, and they break when one hop is unusually slow but still within its own limit.
A circuit breaker opens on your payments dependency during a real incident. Describe what the user sees, what your service does, and how you decide when to close it again.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
An open breaker is a decision about what to do instead, not just a decision to stop calling.
Show a strong answer
What the breaker actually does: after a threshold of failures, it stops calling the dependency and fails immediately. The value is twofold — the caller stops burning resources on calls that will fail, and the struggling dependency stops receiving load it cannot serve.
What the user should see depends on the operation, and this is the design work:
- Taking a payment: a clear, honest failure. "We could not process your payment, nothing has been charged, please try again shortly." Not a spinner, and not a generic 500. The important guarantee is that the message is true: if the request may have reached the provider, the message must not promise nothing was charged.
- Displaying saved cards: degrade. Show the page without them, with a note.
Deciding when to close: never all at once. After a cool-off period the breaker moves to half-open and allows a small number of trial requests through.
- If they succeed, close fully and resume.
- If any fail, reopen and extend the cool-off — usually with exponential growth, so a dependency that is still broken is probed less and less often.
The failure mode to avoid is a breaker that closes fully on one success. The dependency is often recovering, not recovered; slamming the full load back on immediately knocks it over again, and the system oscillates. Half-open with a small trial volume is what makes recovery monotonic rather than cyclical.
One thing to instrument: breaker state changes should be an alert-worthy event. A breaker silently open for a week is a feature nobody noticed was broken.
Key takeaways
- Slow is worse than down: a fast failure releases resources, a slow success holds them until the pool is gone.
- Timeouts are the foundation. Derive them from measured p99, and shrink them as the call chain deepens.
- Retries need idempotency, exponential backoff, jitter and a cap — jitter is the most-omitted and most important.
- Retry storms lengthen outages. Without backoff and a breaker, retrying makes recovery impossible.
- A circuit breaker protects both sides: the caller fails fast, the dependency gets room to recover.
- Bulkheads give each dependency its own pool so one cannot exhaust everything.
- Decide critical versus optional per dependency in advance, and design the degraded response deliberately.
- The patterns compose: bulkhead, breaker, timeout, retry, fallback — in that order.
Further reading
- Michael Nygard, Release It!, second edition (Pragmatic Bookshelf, 2018) — the source for circuit breakers, bulkheads and timeouts.
- Google SRE, Site Reliability Engineering (O'Reilly, 2016), chapter 22 — addressing cascading failures.
- Marc Brooker, "Timeouts, retries and backoff with jitter" (Amazon Builders' Library) — why jitter matters as much as backoff.
- Peter Deutsch and James Gosling, "The Eight Fallacies of Distributed Computing" (Sun Microsystems, 1994–1997).
- Jeffrey Dean and Luiz André Barroso, "The Tail at Scale" (CACM, 2013) — hedged requests as an alternative to plain retries.