Circuit Breaker and Bulkhead Patterns
On this page
Symptom: a payment provider slows down, and the whole site stops
The payment gateway starts taking 30 seconds to time out instead of answering in 300 ms. Checkout threads wait on it. Soon every thread in the checkout service is waiting, so product pages, which share those threads, stop responding too. A problem in one external dependency became a full outage. That's a cascading failure, and two patterns stop it.
How a circuit breaker works: closed, open and half-open
A circuit breaker wraps calls to a dependency and tracks failures. It has three states:
- Closed: normal. Calls go through; failures are counted.
- Open: after too many failures, calls fail immediately without contacting the dependency, for a cooling-off period.
- Half-open: after the cooling-off period, one trial call goes through. Success closes the circuit; failure opens it again.
# cb.py: a minimal circuit breaker (closed -> open -> half-open)
import time
class CircuitOpen(Exception):
pass
class CircuitBreaker:
def __init__(self, failure_threshold=5, reset_after=30.0):
self.failure_threshold, self.reset_after = failure_threshold, reset_after
self.failures, self.state, self.opened_at = 0, "closed", 0.0
def call(self, fn, *args):
if self.state == "open":
if time.monotonic() - self.opened_at < self.reset_after:
raise CircuitOpen("dependency marked down, failing fast")
self.state = "half-open" # let one trial request through
try:
result = fn(*args)
except Exception:
self.failures += 1
if self.state == "half-open" or self.failures >= self.failure_threshold:
self.state, self.opened_at = "open", time.monotonic()
raise
self.failures, self.state = 0, "closed" # success closes the circuit
return resultWith a threshold of 3 against a gateway that keeps timing out, six calls produced:
['timeout', 'timeout', 'timeout', 'fast-fail', 'fast-fail', 'fast-fail'] state: open
... after the reset period, gateway healthy again: 'charged' state: closedThe last three calls failed in microseconds instead of waiting for timeouts. Threads stay free, and the struggling gateway gets breathing room to recover. Production libraries (Polly in .NET, Resilience4j in Java) add failure-rate windows, per-exception rules and metrics; the idea is the same.
In .NET 8 and later, AddStandardResilienceHandler() from Microsoft.Extensions.Http.Resilience puts a total timeout, retries, a circuit breaker and a per-attempt timeout around an HttpClient in one line (docs). Read its defaults before shipping: retries apply to every HTTP method, POST included, unless you call DisableForUnsafeHttpMethods().
Count the right failures. A dependency answering 400 Bad Request or 404 is healthy; the caller sent something wrong. Only timeouts, connection errors and 5xx responses (and usually 429) should push a breaker toward open, or one buggy client can trip the breaker for everyone.
The bulkhead pattern: limit the blast radius
Ships have watertight compartments so one breach doesn't sink the whole hull. A bulkhead gives each dependency its own limited pool of resources: concurrent requests, threads or connections. If payment calls may use at most 20 concurrent slots, a slow payment gateway can tie up those 20, and nothing else. Product pages keep working on their own capacity.
In practice: separate connection pools or HTTP clients per dependency, a concurrency limit per dependency, or separate service instances for critical and non-critical traffic.
In ASP.NET Core, a slow call made with await doesn't hold a thread while it waits, but it still holds memory, a connection and an incoming request slot, so the bulkhead still matters. Code that blocks on async work with .Result or .Wait() is worse: each waiting call pins a thread-pool thread, and a slow dependency can starve the whole pool, freezing endpoints that never call it.
Fallbacks: what to do when a call fails
Failing fast only helps if you have something sensible to do:
- show cached or default data (generic recommendations instead of personalized),
- accept the order and process payment later, with a clear message,
- hide a non-essential feature,
- return a clear error quickly, which is still better than a 30-second hang.
Combining timeouts, retries, circuit breakers and bulkheads
Around each synchronous call, from outside in: a timeout (Day 75) so no call hangs, retries with backoff for transient failures (Day 74), a circuit breaker so retries stop when the dependency is clearly down, and a bulkhead limiting concurrency. Retries without a breaker make an outage worse by multiplying load on a dependency that's already failing.
Check yourself
It fails calls immediately without contacting the dependency. Callers stay responsive, threads aren't tied up waiting, and the failing dependency gets time to recover.
Without it, a breaker would either stay open forever or reopen to full traffic all at once. Half-open lets a single trial request through after a pause: if it succeeds, the breaker closes; if it fails, it opens again. That checks recovery without flooding a dependency that may still be fragile.
It counts client errors as failures. A 400 means the request was bad, not that the dependency is down. Count only timeouts, connection failures and 5xx responses.
A bulkhead: give payment calls their own limited pool of threads or connections. When payment slows down, only that pool fills up and payment calls wait or fail fast, while product pages keep using their own resources. Without isolation, slow calls consume the shared pool everyone needs.