Circuit Breaker wraps a call to something that can fail and remembers how those calls have been going. After enough failures it stops calling at all for a while, failing immediately instead — and then cautiously tries again.
High-level design covers why a system needs one: how a slow dependency exhausts a caller's workers, and how a struggling service under continuous retry pressure cannot recover. This topic is about the pattern itself — its structure, and the four decisions that separate a breaker which helps from one that causes the outage it was meant to prevent.
What it is made of
The structure is two patterns you already have. It is a Proxy — same interface as the thing it wraps, deciding whether the call happens at all — whose behaviour is driven by a State machine.
Diagram, as text
- Caller leads to CircuitBreaker same interface as the subject
- CircuitBreaker same interface as the subject leads to Current state
- Current state leads to call through, record the outcome (Closed)
- Current state leads to fail immediately, subject untouched (Open)
- Current state leads to let a few through, close or reopen on the result (HalfOpen)
- call through, record the outcome leads to Remote dependency
- let a few through, close or reopen on the result leads to Remote dependency
Recognising it as State is not a naming exercise — it is what tells you where the logic goes. Each state answers one question, "may this call proceed?", and decides its own successor. Written as conditionals over a status field, the transition rules scatter across every method, which is precisely the smell the State topic describes.
const Closed = {
name: 'closed',
allows: () => true,
onSuccess: (b) => b.reset(),
onFailure: (b) => (b.failureRate() > b.threshold ? Open : Closed),
};
const Open = {
name: 'open',
allows: (b) => b.clock.now() - b.openedAt >= b.cooldownMs && HalfOpen,
onSuccess: () => Open,
onFailure: () => Open,
};Note b.clock. A breaker's whole behaviour is a function of time, so an injected clock is not a nicety — it is the difference between a test that runs in a millisecond and one that sleeps for the cooldown. This is the Strategy/injection argument arriving somewhere it genuinely pays.
Decision one: what counts as a failure
The most consequential decision in the pattern, and the one no diagram shows.
A breaker counts failures. If it counts the wrong things it either never trips while the dependency is unusable, or trips constantly and takes down a healthy path by itself.
| Outcome | Counts as a failure? |
|---|---|
| Timeout, connection refused, 503 | Yes — the dependency is unwell |
| 500 with no body | Yes |
| 429 rate limited | Yes, and back off — it is asking you to stop |
| 404 not found | No — a correct answer to a question about a missing thing |
| 400 bad request | No — the bug is in the caller; tripping hides it |
| 401 / 403 | No — a credential problem does not heal with a cooldown |
The rule underneath: count failures OF the dependency, not failures OF the request. A breaker that trips on 400s will open under a burst of malformed input and refuse service to every well-formed request behind it — a self-inflicted outage from a component installed to prevent one.
Decision two: what to count over
Trip on a failure rate over a rolling window, with a minimum volume before the rate means anything.
A raw count trips a busy endpoint on its normal background of errors while a quiet one never trips. A rate without a volume floor is worse: one failed call out of one is a 100% failure rate, so a breaker with no minimum opens on the first error after an idle period, every time.
Two numbers, both of which need saying out loud: the window (how much history counts) and the minimum volume (how many calls before the rate is trusted).
Decision three: half-open concurrency
The state everyone draws and few implement carefully.
When the cooldown elapses, the breaker admits trial calls. If it admits all waiting traffic, a service that has just come back up is hit by the full backlog at once and falls over again — and the breaker has converted one outage into an oscillation.
Admit a small fixed number of concurrent trials, and nothing else. Every other caller keeps failing fast until those trials report. One success is usually not enough to close either: a single lucky response says little, and requiring a few consecutive ones costs a moment and avoids flapping.
Decision four: granularity
One breaker per dependency, not one per process and not one per call site.
Too coarse — a single breaker for "the network" — and a failing recommendations service stops payment calls. Too fine — one per URL — and no endpoint sees enough traffic to reach the minimum volume, so none ever trips.
The unit is the thing that fails together: a service, or a distinct endpoint on it with materially different behaviour. This is the same boundary question as an actor's granularity or a partition key, arriving for the fourth time in this module — a recurrence worth noticing.
What an open breaker should return
Failing fast is only useful if the caller can do something with the failure. The options, in order of preference:
- A cached or stale value, where staleness is acceptable — the best outcome, and it needs deciding in advance.
- A reduced response, explicitly marked as degraded so nothing downstream caches or persists it as complete.
- A clear, typed failure the caller can branch on.
What it must not do is return an empty result indistinguishable from a genuine one. That converts a visible outage into silently wrong data — the same argument the Decorator topic makes about a blanket fallback wrapper.
Where the state lives
Per-process by default, and that is usually right: each instance observes its own calls and decides for itself.
Sharing breaker state across a fleet sounds appealing — one instance learns, all benefit — and brings a coordination dependency into the failure path, which is exactly where you least want one. A shared store that is itself unavailable during an incident makes every call slower at the worst moment.
The per-process cost is that each instance must learn independently, so a fleet of two hundred sends two hundred instances' worth of trial traffic. That is usually tolerable. If it is not, the answer is fewer, larger instances or a proxy that owns the breaker for everyone — which is what a sidecar does.
Check your understanding
Common mistakes
Counting client errors as failures. A buggy client release opens the circuit against a healthy dependency.
A raw failure count instead of a rate over a window. Busy endpoints trip on background noise; quiet ones never trip.
A rate with no minimum volume. One failure out of one is 100%, so the breaker opens on the first blip after an idle spell.
Admitting all traffic on half-open. The backlog knocks over the service the moment it recovers.
One breaker for everything, or one per URL. Too coarse couples unrelated dependencies; too fine never reaches the volume floor.
Retry outside the breaker. Each attempt is rejected separately, and the retry loop hammers the service being protected.
Returning an empty result when open. A silent wrong answer instead of a visible failure.
Reading the clock directly. The behaviour is entirely time-dependent, so a test then has to sleep through the cooldown.
Practice
Implement the state machine for a circuit breaker, in words: name every state, every transition, what triggers it, and what data the breaker must hold. Then say how you would test it without sleeping.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
The data it holds is the part that decides whether the transitions can even be evaluated.
Show a strong answer
States and transitions:
- Closed — calls pass through, every outcome is recorded. Transitions to Open when the failure rate over the window exceeds the threshold and the window holds at least the minimum volume. Both conditions, or a single early failure trips it.
- Open — every call fails immediately without touching the dependency. Transitions to Half-open when the cooldown has elapsed. Note this is evaluated on the next call, not by a timer: a background timer wakes a process that may have no traffic, and the state only matters when someone asks.
- Half-open — a bounded number of concurrent trials are admitted; everything else still fails fast. Transitions to Closed after N consecutive successes, back to Open on any failure. A reopen should restart the full cooldown, not a shortened one.
Data it must hold:
- A rolling outcome window — timestamps and success/failure, or bucketed counts. Buckets are the practical choice: a per-call list on a hot path is unbounded memory for a number you could have incremented.
- Current state and when it was entered — the cooldown is measured from the transition.
- Trial counters while half-open — in flight, and consecutive successes.
- An injected clock.
Testing without sleeping — this is the whole reason for the clock:
- Inject a fake clock you advance by hand.
clock.advance(cooldownMs)then assert the next call is admitted. Every timing assertion becomes deterministic and instant. - Test the transitions as a table — from each state, for each outcome, assert the resulting state. That is a dozen cases and it covers the machine exhaustively, which ad-hoc scenario tests do not.
- Test the volume floor explicitly: one failure out of one must not open. This is the case most likely to be wrong and least likely to be written.
- Test half-open concurrency: with the trial limit at two, three simultaneous calls must produce two attempts and one fast failure.
- Test that reopening restarts the cooldown, or a flapping dependency gets retried far too often.
What I would not do: test through a real HTTP client with a stub server. That tests the transport, takes seconds, and leaves the transition table — the actual logic — covered only incidentally.
A team reports that their circuit breaker 'makes things worse': during incidents it opens and stays open long after the dependency recovers, and sometimes opens when the dependency is fine. Diagnose the likely causes and give the order you would fix them.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Two distinct symptoms. Each points at a different one of the four decisions.
Show a strong answer
These are two separate faults and it is worth saying so, because a team experiencing both usually treats them as one flaky component and disables it.
Symptom one — opens when the dependency is fine. Almost certainly counting the wrong outcomes.
- Check what the failure predicate includes. If it counts any non-2xx, then 400s, 401s and 404s are all tripping it. A burst of malformed requests from one buggy client, or a wave of 404s from a crawler hitting dead links, opens the circuit against a perfectly healthy service.
- Also check the volume floor. If it trips on rate alone, a quiet endpoint opens on its first error after an idle period — which looks random and correlates with nothing.
- Fix first, because it is cheap and it is the one destroying trust in the component.
Symptom two — stays open after recovery. The half-open path is not working, and there are three usual causes:
- The cooldown is evaluated by a timer rather than on the next call, and a low-traffic period means nothing wakes it.
- Trials are failing for a reason unrelated to the dependency's health — most often the trial inherits the same aggressive timeout that was failing under load, so a recovered-but-still-warming service never passes.
- Any single trial failure reopens with a fresh, longer cooldown (exponential backoff on the cooldown), so a service that recovers during a long cooldown waits out the whole thing.
Order I would fix them:
- Correct the failure predicate. Stop counting client errors. This alone may remove most of the reported incidents.
- Add or fix the minimum volume floor.
- Make the open→half-open check happen on call arrival, not on a timer.
- Give trial calls a more generous timeout than steady-state calls — a warming service is slow before it is healthy.
- Cap the cooldown so backoff cannot grow unbounded.
What I would add regardless: a metric for state transitions with the reason, and an alert on a breaker open longer than some threshold. This team could not tell why it opened, which is why they ended up debating whether to keep it at all. A breaker that cannot explain itself will eventually be deleted by someone during an incident.
Key takeaways
- Circuit Breaker is a Proxy whose behaviour is a State machine — recognising that is what keeps the transition rules in one place rather than scattered across guard clauses.
- What counts as a failure is the most consequential decision. Count failures of the dependency, never of the request: tripping on 4xx turns a buggy client into a self-inflicted outage.
- Trip on a failure rate over a rolling window, with a minimum volume — a rate over almost no data opens the circuit on a single blip.
- Half-open must admit only a few concurrent trials, or the backlog knocks over the service the moment it recovers.
- One breaker per dependency — the same boundary question as actor granularity and partition keys.
- Breaker outside retry, always; otherwise the retry loop hammers the service being protected.
- An open breaker must return something distinguishable from success — stale, explicitly degraded, or a typed failure.
- Inject the clock. The pattern is entirely time-dependent, and a real clock means tests that sleep.
Further reading
- Michael Nygard, Release It!, second edition (Pragmatic Bookshelf, 2018) — the pattern's original statement, and the failure modes it was written for.
- Martin Fowler, "CircuitBreaker" (2014).
- Betsy Beyer and others, Site Reliability Engineering (O'Reilly, 2016) — retry budgets, cascading failure and recovery.
- Jeffrey Dean and Luiz André Barroso, "The Tail at Scale" (Communications of the ACM, 2013).
- Brendan Burns, Designing Distributed Systems (O'Reilly, 2018) — moving the breaker into a sidecar.