The Retry That Took Down the Service

The incident below is a composite. It never happened exactly like this anywhere, and it has happened approximately like this everywhere. If you have operated a service of any size you will recognise the shape well before the end.

Three services matter. A gateway takes traffic from mobile clients. Behind it sits orders, which cannot render a basket without calling pricing, which reads promotion rules from a small key-value store. All of it had been quiet for months.

09:14 — nothing is failing

A node in the promotions store gets replaced during routine maintenance. Leadership moves, caches are cold, and reads that used to come back in single-digit milliseconds start taking a second or two.

Nothing pages. The error rate is zero. Every request is succeeding; they are simply succeeding slowly, and slow is not a state most alerting is built to notice.

09:17 — timeouts become retries

The pricing client inside orders has a two-second timeout. Requests begin crossing it.

A timeout is not an error the way a 500 is an error. It is an unknown, and the retry policy treats unknowns optimistically: try again, up to three times. Nobody had put a delay between the attempts, because the policy was written during a sprint where the failure being imagined was a dropped packet, and for a dropped packet immediate retry is exactly right.

So each timing-out request instantly becomes three. Load on pricing triples without a single additional user arriving.

09:19 — the degradation spreads

pricing was fine a moment ago. It is not fine now. Its thread pool is fully occupied waiting on a slow dependency, and requests begin queueing at the front door. Once there is a queue, every request is slow — including the ones that never touch a promotion rule and would have returned instantly.

This is the step people miss when they reason about blast radius. The original fault was confined to one store. The retries carried it into a service that had no fault at all.

09:21 — the multiplication

Gateway timeouts are longer, so its own retry layer starts a few minutes behind. It has three attempts too. So does the HTTP client library underneath the pricing call, which was configured by someone else entirely, in a different year.

Three at the gateway, three in orders, three in the client: 3 × 3 × 3 = 27 calls at the bottom of the stack for one user tapping a button once. Every layer is individually defensible. Nobody reviewed the product of them, because the product is not visible in any single file.

09:26 — the restart makes it worse

The first page fires at 09:23, when timeouts finally age into 5xx responses. Someone does the obvious thing and restarts pricing.

This is the moment the incident stops being a slow dependency and starts being a self-sustaining one.

The restart severs every in-flight connection at once. Every client discovers failure in the same instant, and every client's backoff timer starts in that same instant. The instances come back with cold caches and empty connection pools — at their weakest — and the entire synchronised wave lands on them before they have warmed. They fall over in seconds. The team restarts again. Same result, faster.

09:34 — the cause has been gone for twelve minutes

Someone finally graphs the promotions store on its own and finds it healthy since 09:22.

The original fault had been over for twelve minutes and the outage had continued anyway, sustained entirely by the mechanism designed to survive it. Offered load was a function of the failure rate, and the failure rate was a function of offered load. Once that loop's gain passes one, removing the initial cause changes nothing. There are only two exits: less load in, or more capacity — and you cannot scale your way out of a warm-up.

They take the first exit at 09:41, rate limiting hard at the gateway, letting pricing come up under a fraction of real traffic, then ramping over several minutes. Recovered by 09:50, roughly half an hour after the actual defect had repaired itself.

Jitter is the part everyone leaves out

Most teams implement exponential backoff. Very few implement the random component, and the random component is the entire point.

Backoff decides when a single client retries. It does nothing about the fact that a thousand clients all failed within the same few milliseconds and are therefore all running identical timers. Doubling the wait moves the herd; it does not disperse it. You get the same synchronised wave, just later and larger, arriving precisely when the service is most fragile.

Jitter is the only ingredient that decorrelates independent clients from each other. Sleeping a random duration up to the backoff cap, rather than the cap itself, turns a wall of traffic into a ramp — and a ramp is something a recovering service can actually absorb.

The reframing

The fix that stuck was not another retry rule. It was a retry budget: a client may spend no more than a small fraction of its total request volume on retries, enforced as a counter, so amplification is bounded no matter how many layers each add independently. Underneath that, a circuit breaker to stop trying at all once failure is the norm, and retries removed from every layer except the one close enough to the user to make a real decision.

But the change in thinking mattered more than any of them. A retry policy is not error handling. It is a small program, running on every client simultaneously, that generates traffic in proportion to your failure rate — at exactly the moment you have the least capacity to serve it. Budget it like capacity, review it like capacity, and load-test it with the dependency made slow rather than made dead. Slow is the case that breaks you.