Catalog My learning Blog
Distributed Systems Explained: CAP, Clocks and Replication · Phase 4 · Lesson 24 · 6 min read

Cascading Failures: Backpressure and Circuit Breakers

On this page
  1. How a cascading failure spreads
  2. Timeouts and retries: the first defense
  3. Backpressure and load shedding
  4. How a circuit breaker works
  5. The bulkhead pattern
  6. Check yourself

How a cascading failure spreads

The payments service slows from 50 ms to 10 seconds per call. Checkout calls payments and waits. Each waiting request holds a thread and a connection. Within a minute, checkout's thread pool is full, so it stops answering, including requests that never needed payments. Now the cart service, which calls checkout, fills up too. Retries from every layer multiply the load on payments, which keeps it slow.

That's a cascading failure: the failure travels upstream through waiting callers. It rarely starts with a crash; it usually starts with slowness, because a dead service fails fast and a slow one holds resources.

Timeouts and retries: the first defense

Every network call needs a timeout, set from the dependency's normal latency (for example a little above its p99), not a library default of 30 seconds or infinity. Within a request chain, pass a deadline downstream (gRPC does this natively) so lower layers stop working on requests the caller has already given up on.

Retries need limits, exponential backoff and jitter, and ideally a retry budget, so they don't turn a slowdown into an overload. Each one does a specific job:

  • Exponential backoff: wait longer after each failure (100 ms, then 200, 400, 800), up to a cap.
  • Jitter: wait a random time between zero and that ceiling ("full jitter"). Without it, every client that failed at the same moment retries at the same moment, and the retries hit the struggling service as one wave.
  • A retry budget: retry only while retries stay under a fixed share of normal traffic, for example 10 percent.
  • Retries at one layer only. If the browser, the gateway, checkout and the payments client each try 3 times, one click can become 3 × 3 × 3 × 3 = 81 calls to payments.
Python
# retry.py: capped exponential backoff with full jitter, plus a retry budget
import random, time

class RetryBudget:
    """Allow retries up to `ratio` of first attempts, like a token bucket."""
    def __init__(self, ratio=0.1, min_tokens=10):
        self.ratio, self.tokens, self.max = ratio, min_tokens, min_tokens
    def on_request(self):  self.tokens = min(self.max, self.tokens + self.ratio)
    def can_retry(self):
        if self.tokens >= 1:
            self.tokens -= 1
            return True
        return False

def call_with_retries(fn, budget, attempts=4, base=0.1, cap=2.0, sleep=time.sleep):
    budget.on_request()
    for attempt in range(attempts):
        try:
            return fn()
        except TimeoutError:
            if attempt == attempts - 1 or not budget.can_retry():
                raise
            delay = random.uniform(0, min(cap, base * 2 ** attempt))   # full jitter
            sleep(delay)

class Counted:
    def __init__(self): self.calls = 0
    def __call__(self):
        self.calls += 1
        raise TimeoutError                      # a full outage: every call times out

for label, budget in (("unlimited retries", RetryBudget(ratio=1, min_tokens=10**9)),
                      ("10 percent budget", RetryBudget())):
    dep = Counted()
    for _ in range(1000):
        try: call_with_retries(dep, budget, sleep=lambda s: None)   # skip real sleeps here
        except TimeoutError: pass
    print(f"{label}: 1000 requests -> {dep.calls} calls to the failing dependency")
Code
unlimited retries: 1000 requests -> 4000 calls to the failing dependency
10 percent budget: 1000 requests -> 1109 calls to the failing dependency

During a full outage, unlimited retries quadruple the load on the service that's already failing. With the budget, retries stop once its tokens run out, and the dependency sees barely more than the original traffic. When it's only a brief blip, the budget still has tokens and retries work as usual. gRPC's retry throttling and Finagle's retry budgets work this way, and the AWS SDKs use backoff with jitter by default.

Backpressure and load shedding

Backpressure means a component signals "slow down" instead of accepting work it can't finish: bounded queues, 429 Too Many Requests or 503 with Retry-After, TCP flow control, reactive streams' demand signals. An unbounded queue hides overload until memory runs out, and every request in it waits longer than its caller cares about.

Load shedding is the server's version: when overloaded, reject some requests early and cheaply (lowest priority first) so the rest succeed. Serving 80 percent of requests well beats serving 100 percent of them too slowly to matter.

How a circuit breaker works

A circuit breaker wraps calls to a dependency and stops calling it once it's clearly failing, so callers fail fast instead of waiting.

  • Closed: calls go through; failures are counted.
  • Open: after a threshold of failures, calls are rejected immediately for a cooldown period.
  • Half-open: after the cooldown, one trial call goes through. Success closes the circuit; failure opens it again.
A circuit breaker opens after repeated failures, fails fast during a cooldown, then lets a trial request through in half-open state to decide whether to close again.
A circuit breaker opens after repeated failures, fails fast during a cooldown, then lets a trial request through in half-open state to decide whether to close again.
Python
# breaker.py: a minimal circuit breaker (closed -> open -> half-open)
import time

class CircuitOpen(Exception): pass

class CircuitBreaker:
    def __init__(self, threshold=3, cooldown=5.0, clock=time.monotonic):
        self.threshold, self.cooldown, self.clock = threshold, cooldown, clock
        self.failures, self.state, self.opened_at = 0, "closed", 0.0

    def call(self, fn):
        if self.state == "open":
            if self.clock() - self.opened_at < self.cooldown:
                raise CircuitOpen("failing fast")
            self.state = "half-open"            # let one trial request through
        try:
            result = fn()
        except Exception:
            self.failures += 1
            if self.state == "half-open" or self.failures >= self.threshold:
                self.state, self.opened_at = "open", self.clock()
            raise
        self.failures, self.state = 0, "closed"
        return result

Simulating payments timing out from t=0 and recovering at t=8, with a threshold of 3 and a 5-second cooldown:

Config
t=0s  error: timeout         state=closed
t=1s  error: timeout         state=closed
t=2s  error: timeout         state=open
t=3s  rejected: failing fast state=open
t=4s  rejected: failing fast state=open
t=8s  ok                     state=closed
t=9s  ok                     state=closed

At t=3 and t=4, callers got an instant error instead of waiting on a timeout, and payments got time to recover. In production, use a library (Resilience4j, Polly, Envoy's outlier detection) that tracks failure rates over a sliding window and is safe across threads.

The bulkhead pattern

Ships have watertight compartments so one leak doesn't sink the vessel. A bulkhead gives each dependency its own limited pool of threads or connections. When payments is slow, only the payments pool fills; requests that don't touch payments keep working.

Check yourself

A crashed service fails fast, while a slow one holds callers' threads and connections, which fills their pools and spreads the failure upstream.

After the cooldown, it lets a limited number of trial requests through to test the dependency. If they succeed, the circuit closes and normal traffic resumes; if they fail, it opens again for another cooldown, protecting the dependency while it recovers.

Latency grows without limit, callers time out on requests still queued, and memory may run out; a bounded queue with rejection gives backpressure.

Without it, clients that failed at the same moment all retry at the same moment, so the retries arrive as a synchronized wave that can knock the recovering service over again.

Giving each dependency its own limited pool of threads or connections, so one slow dependency can exhaust only its pool rather than every resource in the service.