Catalog My learning Blog
Distributed Systems Explained: CAP, Clocks and Replication · Phase 4 · Lesson 25 · 4 min read

Graceful Degradation in Distributed Systems

On this page
  1. Partial outages are the normal case
  2. Rank dependencies by criticality
  3. Fallback strategies
  4. Degrading on purpose: kill switches and brownouts
  5. Testing degraded modes
  6. Check yourself

Partial outages are the normal case

With dozens of dependencies, something is usually degraded somewhere. The question isn't whether the recommendation service will fail, but what the checkout page does when it does. Graceful degradation means the system keeps delivering its core function, possibly with fewer features, instead of failing entirely.

Rank dependencies by criticality

List what each page or API depends on, and label each dependency:

Dependency of checkoutCriticalityWhen it fails
PaymentsCriticalCan't complete the order; show a clear error and keep the cart
Inventory checkImportantAccept the order and confirm stock asynchronously, within limits
Shipping estimatesOptionalShow "estimated at confirmation"
RecommendationsOptionalShow a cached or generic list, or hide the section
Reviews widgetOptionalHide it

The rule: a failure in an optional dependency must never fail a critical path. That sounds obvious, and is often broken by a shared thread pool, a missing timeout, or an unhandled exception in a widget.

Checkout keeps its essential payment and confirmation calls while a failing recommendation service is skipped by a circuit breaker and replaced with a fallback list.
Checkout keeps its essential payment and confirmation calls while a failing recommendation service is skipped by a circuit breaker and replaced with a fallback list.

Fallback strategies

  • Cached or stale data: serve the last known good value. Stale prices for a product page might be fine; stale balances for payments aren't.
  • Defaults: a generic recommendation list, a standard shipping estimate.
  • Reduced functionality: search without personalization, read-only mode during a database failover.
  • Deferred work: accept the request, queue it and process it later (emails, analytics, some inventory updates).
  • Omit the feature: hide the section rather than show an error.

Combine fallbacks with the protections from Day 24: timeouts keep a slow optional call from delaying the page, circuit breakers skip a failing dependency quickly, and bulkheads keep its failures in their own pool. In code, a timeout, a bulkhead and a fallback fit in a few lines:

Python
# fallback.py: an optional dependency gets a short timeout and a fallback
import time
from concurrent.futures import ThreadPoolExecutor

recs_pool = ThreadPoolExecutor(max_workers=4)     # a bulkhead: recommendations get 4 threads, no more
POPULAR = ["best sellers"]                        # a generic list, refreshed by a background job
fallbacks_served = 0

def recommendations_service(user_id):
    time.sleep(3)                                 # the dependency is having a bad day
    return [f"picked for {user_id}"]

def recommendations(user_id):
    global fallbacks_served
    future = recs_pool.submit(recommendations_service, user_id)
    try:
        return future.result(timeout=0.2)         # optional: wait 200 ms at most
    except Exception:                             # timeout or error: never fail the page
        fallbacks_served += 1                     # export this as a metric
        return POPULAR

def product_page(user_id):
    start = time.monotonic()
    page = {"product": "kettle", "price": 39, "recommended": recommendations(user_id)}
    return page, round(time.monotonic() - start, 2)

print(product_page("u1"))
print("fallbacks served:", fallbacks_served)
Code
({'product': 'kettle', 'price': 39, 'recommended': ['best sellers']}, 0.2)
fallbacks served: 1

The page came back in 200 ms with the generic list while the recommendations call was still running in its own pool. That pool has 4 threads, so a flood of slow calls ties up those 4 and nothing else. One gap to close in production: extra work submitted to a full pool waits in its queue without limit, so also reject new calls when too many are already waiting, or put a circuit breaker in front.

Degrading on purpose: kill switches and brownouts

Sometimes you degrade before anything fails, to protect the core under load:

  • Feature flags or kill switches turn off expensive features instantly (personalized ranking, high-resolution images, live counters) without deploying.
  • Brownouts: shed optional work progressively as load rises, starting with the least valuable.
  • Priority-based load shedding: keep checkout and login, drop background syncs and prefetches first.

Decide these levels in advance and document who can flip them. During an incident is the wrong time to discover which flags exist.

Testing degraded modes

Degraded modes that are never exercised rot. Fallback code paths break silently, caches expire, and a "hide the widget" path throws an exception nobody noticed.

  • Test fallbacks in CI by making dependencies fail or time out in integration tests.
  • Run game days or chaos experiments: inject failures and latency into staging, then carefully into production (Netflix's Chaos Monkey popularized this).
  • Monitor fallback rates: a fallback being served is a signal; a sudden rise means a dependency is in trouble, even if users haven't noticed.
  • Tell users plainly when something is limited ("Recommendations are unavailable right now") rather than showing broken UI.

Check yourself

Map each feature's dependencies and rank them by criticality, so you know which failures must be absorbed and which must stop the request.

A short timeout and a fallback for an optional dependency, such as showing best-sellers, and isolation so its failure can't break the critical path. Recommendations are nice to have; the product page itself must load.

Fallbacks hide failures from users, which is their purpose, but also from you. A rising fallback rate reveals a failing dependency early, while the page still works, and shows how much of the experience is degraded.

A runtime flag that turns off a non-essential feature instantly, without a deploy, to shed load or remove a failing dependency during an incident.

Inject failures in staging or controlled production experiments, such as blocking a dependency or adding latency, and verify the fallback works and alerts fire.