Catalog My learning Blog
System Design Roadmap: 100-Day Course for Engineers · Phase 8 · Lesson 75 · 5 min read

Timeouts and Failure Isolation in System Design

On this page
  1. Why default timeouts cause cascading failures
  2. How to choose a timeout
  3. Deadline propagation, not just timeouts
  4. Failure isolation: timeouts, bulkheads and circuit breakers
  5. How a cascade happens, and where each pattern breaks it
  6. Check yourself

Why default timeouts cause cascading failures

An HTTP client's default timeout is often far longer than anyone expects: 100 seconds for .NET's HttpClient, and some libraries wait forever by default. When a dependency hangs instead of failing, every call waits that long, threads and connections pile up, and the caller runs out of capacity long before the timeout fires. A missing or oversized timeout is one of the most common causes of cascading failure.

How to choose a timeout

Base it on the dependency's measured latency, not a round number:

  • Look at its p99 and p99.9 under normal load (Day 3).
  • Set the timeout somewhat above that: if p99 is 120 ms and p99.9 is 300 ms, a 500 ms timeout catches real hangs without cutting off normal slow requests.
  • Separate the connect timeout (short, often a second or less) from the request timeout.

Review timeouts when latency changes. A timeout set when the dependency was fast may cut off legitimate requests later, or one set generously may be useless.

Databases need timeouts too, and their defaults are generous. SqlClient and Npgsql both default to a 30-second command timeout, so a query stuck behind a lock can hold a request and a pooled connection for half a minute. SQL Server's lock timeout defaults to waiting forever. Set command timeouts from measured query latency, especially on request paths.

ASP.NET Core can also put a ceiling on the whole request: the request timeouts middleware (.NET 8 and later) cancels the request's CancellationToken after a limit you set per endpoint. Pass that token down to every database call and HTTP call, or the timeout fires and the work carries on regardless.

Deadline propagation, not just timeouts

A user request with a 2-second budget calls service A, which calls B, which calls C. If each layer has its own 2-second timeout, A can still be waiting on B long after the user's request has given up, doing work nobody will see.

Propagate a deadline: the absolute time by which the whole request must finish. Each service passes the remaining time to the next and uses it as the upper bound for its own calls. gRPC does this natively with deadlines; with HTTP you pass a header and use cancellation tokens (CancellationToken in .NET, context.Context in Go). When the deadline passes, everyone stops.

Code
user request deadline: 2,000 ms
  gateway   spends 20 ms  -> calls A with 1,980 ms left
  service A spends 300 ms -> calls B with ~1,650 ms left (minus a margin)
  service B                -> checks remaining time; if too little, fails fast instead of starting work

Failure isolation: timeouts, bulkheads and circuit breakers

A bulkhead gives each dependency its own resource pool, and a circuit breaker stops calling a dependency after too many failures.
A bulkhead gives each dependency its own resource pool, and a circuit breaker stops calling a dependency after too many failures.
  • Timeouts stop any single call from hanging indefinitely.
  • Bulkheads (Day 56) cap how many concurrent calls go to each dependency, so a slow one can only tie up its own slots.
  • Circuit breakers (Day 56) stop calling a dependency that keeps failing, so callers fail fast and the dependency can recover.

Retries (Day 74) sit inside the deadline and are stopped by the circuit breaker.

One more tool for tail latency rather than failure: hedged requests. If a read hasn't answered by its usual p95 time, send the same request to a second replica and use whichever answers first. It costs a few percent more load and cuts the slow tail sharply, which is why Google's "The Tail at Scale" recommends it (Day 3). Use it only for idempotent reads.

How a cascade happens, and where each pattern breaks it

  1. Recommendations becomes slow.
  2. Product pages wait on it; a timeout caps each wait at 300 ms.
  3. Many waits still pile up; a bulkhead limits recommendation calls to 20 concurrent, so product pages' other work keeps its capacity.
  4. Recommendations keeps failing; a circuit breaker opens and calls fail instantly.
  5. Product pages render with a fallback: bestsellers instead of personalized picks.

Without these, step 1 becomes "the whole site is down".

Check yourself

A few hundred milliseconds, somewhat above the observed p99.9, so real hangs are cut off without failing normal slow requests.

Downstream calls should stop when the original request has given up. A shrinking deadline prevents work nobody is waiting for.

A bulkhead, which caps the number of concurrent calls to that dependency with its own limited pool or semaphore, combined with a timeout on each call. When the dependency slows down, only its compartment fills and its calls fail fast, while the rest of the service keeps its threads.

The request's CancellationToken isn't passed to the database calls, so they run to their own 30-second command timeout. Pass the token down and set command timeouts from measured latency.

Many defaults are very long or infinite, such as 100 seconds for .NET's HttpClient. During an incident, requests hang that long, holding threads and connections, and the caller exhausts its resources long before the timeout fires. Set explicit timeouts based on each dependency's real latency.