Graceful Degradation in Distributed Systems
On this page
Partial outages are the normal case
With dozens of dependencies, something is usually degraded somewhere. The question isn't whether the recommendation service will fail, but what the checkout page does when it does. Graceful degradation means the system keeps delivering its core function, possibly with fewer features, instead of failing entirely.
Rank dependencies by criticality
List what each page or API depends on, and label each dependency:
| Dependency of checkout | Criticality | When it fails |
|---|---|---|
| Payments | Critical | Can't complete the order; show a clear error and keep the cart |
| Inventory check | Important | Accept the order and confirm stock asynchronously, within limits |
| Shipping estimates | Optional | Show "estimated at confirmation" |
| Recommendations | Optional | Show a cached or generic list, or hide the section |
| Reviews widget | Optional | Hide it |
The rule: a failure in an optional dependency must never fail a critical path. That sounds obvious, and is often broken by a shared thread pool, a missing timeout, or an unhandled exception in a widget.
Fallback strategies
- Cached or stale data: serve the last known good value. Stale prices for a product page might be fine; stale balances for payments aren't.
- Defaults: a generic recommendation list, a standard shipping estimate.
- Reduced functionality: search without personalization, read-only mode during a database failover.
- Deferred work: accept the request, queue it and process it later (emails, analytics, some inventory updates).
- Omit the feature: hide the section rather than show an error.
Combine fallbacks with the protections from Day 24: timeouts keep a slow optional call from delaying the page, circuit breakers skip a failing dependency quickly, and bulkheads keep its failures in their own pool. In code, a timeout, a bulkhead and a fallback fit in a few lines:
# fallback.py: an optional dependency gets a short timeout and a fallback
import time
from concurrent.futures import ThreadPoolExecutor
recs_pool = ThreadPoolExecutor(max_workers=4) # a bulkhead: recommendations get 4 threads, no more
POPULAR = ["best sellers"] # a generic list, refreshed by a background job
fallbacks_served = 0
def recommendations_service(user_id):
time.sleep(3) # the dependency is having a bad day
return [f"picked for {user_id}"]
def recommendations(user_id):
global fallbacks_served
future = recs_pool.submit(recommendations_service, user_id)
try:
return future.result(timeout=0.2) # optional: wait 200 ms at most
except Exception: # timeout or error: never fail the page
fallbacks_served += 1 # export this as a metric
return POPULAR
def product_page(user_id):
start = time.monotonic()
page = {"product": "kettle", "price": 39, "recommended": recommendations(user_id)}
return page, round(time.monotonic() - start, 2)
print(product_page("u1"))
print("fallbacks served:", fallbacks_served)({'product': 'kettle', 'price': 39, 'recommended': ['best sellers']}, 0.2)
fallbacks served: 1The page came back in 200 ms with the generic list while the recommendations call was still running in its own pool. That pool has 4 threads, so a flood of slow calls ties up those 4 and nothing else. One gap to close in production: extra work submitted to a full pool waits in its queue without limit, so also reject new calls when too many are already waiting, or put a circuit breaker in front.
Degrading on purpose: kill switches and brownouts
Sometimes you degrade before anything fails, to protect the core under load:
- Feature flags or kill switches turn off expensive features instantly (personalized ranking, high-resolution images, live counters) without deploying.
- Brownouts: shed optional work progressively as load rises, starting with the least valuable.
- Priority-based load shedding: keep checkout and login, drop background syncs and prefetches first.
Decide these levels in advance and document who can flip them. During an incident is the wrong time to discover which flags exist.
Testing degraded modes
Degraded modes that are never exercised rot. Fallback code paths break silently, caches expire, and a "hide the widget" path throws an exception nobody noticed.
- Test fallbacks in CI by making dependencies fail or time out in integration tests.
- Run game days or chaos experiments: inject failures and latency into staging, then carefully into production (Netflix's Chaos Monkey popularized this).
- Monitor fallback rates: a fallback being served is a signal; a sudden rise means a dependency is in trouble, even if users haven't noticed.
- Tell users plainly when something is limited ("Recommendations are unavailable right now") rather than showing broken UI.
Check yourself
Map each feature's dependencies and rank them by criticality, so you know which failures must be absorbed and which must stop the request.
A short timeout and a fallback for an optional dependency, such as showing best-sellers, and isolation so its failure can't break the critical path. Recommendations are nice to have; the product page itself must load.
Fallbacks hide failures from users, which is their purpose, but also from you. A rising fallback rate reveals a failing dependency early, while the page still works, and shows how much of the experience is degraded.
A runtime flag that turns off a non-essential feature instantly, without a deploy, to shed load or remove a failing dependency during an incident.
Inject failures in staging or controlled production experiments, such as blocking a dependency or adding latency, and verify the fallback works and alerts fire.