Catalog My learning Blog
System Design Roadmap: 100-Day Course for Engineers · Phase 6 · Lesson 56 · 5 min read

Circuit Breaker and Bulkhead Patterns

On this page
  1. Symptom: a payment provider slows down, and the whole site stops
  2. How a circuit breaker works: closed, open and half-open
  3. The bulkhead pattern: limit the blast radius
  4. Fallbacks: what to do when a call fails
  5. Combining timeouts, retries, circuit breakers and bulkheads
  6. Check yourself

Symptom: a payment provider slows down, and the whole site stops

The payment gateway starts taking 30 seconds to time out instead of answering in 300 ms. Checkout threads wait on it. Soon every thread in the checkout service is waiting, so product pages, which share those threads, stop responding too. A problem in one external dependency became a full outage. That's a cascading failure, and two patterns stop it.

How a circuit breaker works: closed, open and half-open

A circuit breaker wraps calls to a dependency and tracks failures. It has three states:

  • Closed: normal. Calls go through; failures are counted.
  • Open: after too many failures, calls fail immediately without contacting the dependency, for a cooling-off period.
  • Half-open: after the cooling-off period, one trial call goes through. Success closes the circuit; failure opens it again.
Python
# cb.py: a minimal circuit breaker (closed -> open -> half-open)
import time

class CircuitOpen(Exception):
    pass

class CircuitBreaker:
    def __init__(self, failure_threshold=5, reset_after=30.0):
        self.failure_threshold, self.reset_after = failure_threshold, reset_after
        self.failures, self.state, self.opened_at = 0, "closed", 0.0

    def call(self, fn, *args):
        if self.state == "open":
            if time.monotonic() - self.opened_at < self.reset_after:
                raise CircuitOpen("dependency marked down, failing fast")
            self.state = "half-open"                  # let one trial request through
        try:
            result = fn(*args)
        except Exception:
            self.failures += 1
            if self.state == "half-open" or self.failures >= self.failure_threshold:
                self.state, self.opened_at = "open", time.monotonic()
            raise
        self.failures, self.state = 0, "closed"       # success closes the circuit
        return result

With a threshold of 3 against a gateway that keeps timing out, six calls produced:

Code
['timeout', 'timeout', 'timeout', 'fast-fail', 'fast-fail', 'fast-fail']  state: open
... after the reset period, gateway healthy again: 'charged'               state: closed

The last three calls failed in microseconds instead of waiting for timeouts. Threads stay free, and the struggling gateway gets breathing room to recover. Production libraries (Polly in .NET, Resilience4j in Java) add failure-rate windows, per-exception rules and metrics; the idea is the same.

In .NET 8 and later, AddStandardResilienceHandler() from Microsoft.Extensions.Http.Resilience puts a total timeout, retries, a circuit breaker and a per-attempt timeout around an HttpClient in one line (docs). Read its defaults before shipping: retries apply to every HTTP method, POST included, unless you call DisableForUnsafeHttpMethods().

Count the right failures. A dependency answering 400 Bad Request or 404 is healthy; the caller sent something wrong. Only timeouts, connection errors and 5xx responses (and usually 429) should push a breaker toward open, or one buggy client can trip the breaker for everyone.

Circuit breaker states closed, open and half-open, next to bulkheads giving each dependency its own isolated resource pool.
Circuit breaker states closed, open and half-open, next to bulkheads giving each dependency its own isolated resource pool.

The bulkhead pattern: limit the blast radius

Ships have watertight compartments so one breach doesn't sink the whole hull. A bulkhead gives each dependency its own limited pool of resources: concurrent requests, threads or connections. If payment calls may use at most 20 concurrent slots, a slow payment gateway can tie up those 20, and nothing else. Product pages keep working on their own capacity.

In practice: separate connection pools or HTTP clients per dependency, a concurrency limit per dependency, or separate service instances for critical and non-critical traffic.

In ASP.NET Core, a slow call made with await doesn't hold a thread while it waits, but it still holds memory, a connection and an incoming request slot, so the bulkhead still matters. Code that blocks on async work with .Result or .Wait() is worse: each waiting call pins a thread-pool thread, and a slow dependency can starve the whole pool, freezing endpoints that never call it.

Fallbacks: what to do when a call fails

Failing fast only helps if you have something sensible to do:

  • show cached or default data (generic recommendations instead of personalized),
  • accept the order and process payment later, with a clear message,
  • hide a non-essential feature,
  • return a clear error quickly, which is still better than a 30-second hang.

Combining timeouts, retries, circuit breakers and bulkheads

Around each synchronous call, from outside in: a timeout (Day 75) so no call hangs, retries with backoff for transient failures (Day 74), a circuit breaker so retries stop when the dependency is clearly down, and a bulkhead limiting concurrency. Retries without a breaker make an outage worse by multiplying load on a dependency that's already failing.

Check yourself

It fails calls immediately without contacting the dependency. Callers stay responsive, threads aren't tied up waiting, and the failing dependency gets time to recover.

Without it, a breaker would either stay open forever or reopen to full traffic all at once. Half-open lets a single trial request through after a pause: if it succeeds, the breaker closes; if it fails, it opens again. That checks recovery without flooding a dependency that may still be fragile.

It counts client errors as failures. A 400 means the request was bad, not that the dependency is down. Count only timeouts, connection failures and 5xx responses.

A bulkhead: give payment calls their own limited pool of threads or connections. When payment slows down, only that pool fills up and payment calls wait or fail fast, while product pages keep using their own resources. Without isolation, slow calls consume the shared pool everyone needs.