Load Balancer Health Checks and Failover
On this page
Learning Objective
Distinguish active from passive health checking, explain why a shallow health check can report success while a backend is completely unable to serve real traffic, and describe the ejection, circuit breaking, and connection draining mechanisms that let a fleet route around failure without amplifying it.
Concept Explanation
A load balancer is only as good as its model of which backends are healthy, and that model is always an approximation built from indirect evidence. Active health checking means the balancer independently probes each backend on an interval, typically an HTTP GET to a known path or a TCP connect, and marks it unhealthy after some number of consecutive failures. Passive health checking, also called outlier detection, means the balancer draws conclusions from the real traffic it is already sending: if a backend returns 5xx responses or times out at a rate that diverges sharply from its peers, it gets ejected from the pool for a cooldown period. These are complementary rather than competing, and mature systems run both, because each catches a class of failure the other misses.
The single most common health check mistake is making it too shallow. A handler that returns 200 OK unconditionally proves only that the process is running and the HTTP stack is listening. It says nothing about whether the database connection pool is exhausted, whether a required downstream dependency is unreachable, whether the JVM is in a garbage collection death spiral, or whether the disk holding the write-ahead log is full. A backend in any of those states will fail every real request while passing every health check, so the load balancer will keep sending it traffic indefinitely. The fix is a deep health check that actually exercises the critical path: acquire a connection from the pool, run a trivial query, confirm the cache client is connected, and return 503 if any of it fails.
There is a serious counterweight to that advice, though, and it is the reason deep health checks are dangerous when done naively. If every backend health check verifies the shared database, then a brief database hiccup causes every backend to simultaneously report unhealthy, the load balancer ejects the entire pool, and a degradation that would have affected some requests becomes a total outage with zero capacity serving anything. The industry pattern that addresses this is fail open, sometimes called panic mode or minimum healthy percentage: when the fraction of healthy backends drops below a configured threshold, typically somewhere around fifty percent, the balancer stops honoring health status entirely and distributes across all backends, on the reasoning that a partially working fleet is strictly better than no fleet. Envoy implements this as its panic threshold and AWS implements a similar behavior for its target groups.
Ejection needs hysteresis in both directions. Removing a backend after a single failed probe makes the system twitchy and turns transient network blips into capacity loss, so balancers require N consecutive failures before ejection. Restoring a backend immediately after one successful probe is worse, because a machine recovering from overload will pass a single cheap probe, receive its full share of traffic instantly, and immediately fall over again, producing a flapping cycle. Requiring several consecutive successes before full restoration, and ideally ramping traffic back up gradually rather than all at once, is what prevents the recovery itself from becoming the next failure. Slow start, where a newly healthy backend receives a linearly increasing share of traffic over some warm-up window, exists for exactly this reason and matters especially for runtimes with just-in-time compilation or cold caches.
Finally, planned removal deserves different handling from failure. Connection draining, also called graceful shutdown or deregistration delay, means the balancer stops sending new requests to a backend but allows in-flight requests to complete for a bounded period before the instance is terminated. Skipping this is how a routine deployment produces a burst of client-visible errors that looks exactly like an outage in the dashboards. The correct sequence is to mark the instance as draining, wait for the load balancer to observe that and stop routing, let outstanding requests finish or time out, and only then terminate the process.
| Mechanism | What It Detects | What It Misses | Tuning Risk |
|---|---|---|---|
| Active probe (shallow) | Process down, port closed, process hung | Every dependency failure; the process happily returns 200 while failing real work | Almost none, which is why it gives false confidence |
| Active probe (deep) | Dependency failures, exhausted pools, degraded internals | Failures specific to particular request types or tenants | A shared dependency failing marks the entire fleet unhealthy at once |
| Passive outlier detection | Real error rates and latency on actual production traffic | A backend receiving no traffic yet, such as one just added | Too aggressive an ejection threshold removes healthy capacity during a traffic spike |
| Panic / fail-open threshold | Nothing; it is a safety valve for the above | N/A | Set too high and it disables health checking during normal partial failure |
Worked Example: A Health Endpoint That Is Deep Enough but Not Fragile
The goal is a check that catches real local failure without turning a shared dependency wobble into a fleet-wide ejection. The pattern below separates liveness, which asks whether this process should be restarted, from readiness, which asks whether this process should receive traffic, and it deliberately treats a shared dependency differently from a local one.
// /healthz/live - liveness. Answer only: is this process wedged?
// Must NOT check dependencies. If it fails, the orchestrator restarts us,
// and restarting will not fix someone else's database.
app.get('/healthz/live', (req, res) => res.status(200).send('ok'));
// /healthz/ready - readiness. Answer: should the LB send me traffic?
app.get('/healthz/ready', async (req, res) => {
const checks = {};
// LOCAL state: if these fail, this instance specifically is broken and
// ejecting it genuinely helps, because its peers are probably fine.
checks.dbPool = dbPool.availableCount() > 0;
checks.heapOk = process.memoryUsage().heapUsed < HEAP_EJECT_THRESHOLD;
checks.warmedUp = cacheClient.isConnected() && routesLoaded;
// SHARED dependency: check it, but degrade rather than eject. If the
// shared DB is down, every instance would fail this simultaneously and
// the whole pool would be removed - turning degradation into an outage.
checks.sharedDbReachable = await pingSharedDb({ timeoutMs: 200 });
const localOk = checks.dbPool && checks.heapOk && checks.warmedUp;
if (!localOk) return res.status(503).json(checks);
// Shared dependency down: still report ready, serve what we can
// (cached reads, queued writes), and let the panic threshold and the
// application's own circuit breaker handle the blast radius.
res.status(200).json(checks);
});
// Load balancer config expressing hysteresis in both directions:
// interval: 2s
// unhealthy_threshold: 3 (6s to eject - tolerates a blip)
// healthy_threshold: 5 (10s to restore - prevents flapping)
// slow_start_window: 30s (ramp traffic back linearly, not instantly)
// panic_threshold: 50% (below this, ignore health and use everyone)
// deregistration_delay: 30s (drain in-flight work before termination)Real-World Examples
- Amazon documents the fail-open behavior explicitly in its Builders Library article on implementing health checks, describing how a health check that is too aggressive can cause a whole service to be taken out of rotation over a dependency problem, and how Elastic Load Balancing responds to all targets in a target group being marked unhealthy by routing to all of them anyway rather than failing every request. Amazon frames this as choosing a partially working system over a completely dead one, which is the same reasoning behind Envoy panic thresholds.
- Kubernetes formalizes the liveness and readiness split that grew out of exactly the failure described in this lesson: a liveness probe failure restarts the container, while a readiness probe failure only removes the pod from Service endpoints. The Kubernetes documentation warns specifically against checking downstream dependencies in a liveness probe, because a shared dependency outage would then trigger a cascading restart of every pod in the deployment, converting a recoverable degradation into a full crash loop across the fleet.
Review Questions & Answers
Passive health checking, or outlier detection, is missing: nothing is drawing conclusions from the actual production traffic, so the balancer only knows what its synthetic probe tells it. Deepening the active probe helps for the specific dependency you thought to check, but it is fundamentally a guess about which failure modes matter, and it will always miss the ones you did not anticipate, such as a failure that only affects requests for a particular tenant or a particular code path. Outlier detection is the more general mechanism because it observes real outcomes rather than a proxy for them. The complete answer runs both: an active probe catches a backend that has stopped receiving traffic entirely, and passive detection catches a backend that is receiving traffic and failing it.
Because the two probes trigger completely different remediations. A readiness probe failure removes the pod from service endpoints, which is reversible and appropriate when the pod genuinely cannot serve. A liveness probe failure restarts the container, which is only a sensible remediation for a problem restarting can actually fix, such as a deadlocked process. If a liveness probe checks a shared database and that database becomes briefly unreachable, every pod in the deployment fails liveness simultaneously and every pod restarts simultaneously, destroying all warm caches and connection pools and producing a thundering herd of reconnections against a database that was already struggling. The restart makes the situation strictly worse while fixing nothing.
The two directions have asymmetric costs. Ejecting a healthy backend briefly costs a little capacity and is easily recovered, so being somewhat quick to eject is cheap. Restoring a backend that is not actually ready is expensive, because it immediately receives its full share of traffic, fails it, and gets ejected again, producing a flapping cycle that both wastes requests and destabilizes the load distribution across the rest of the pool. A machine recovering from an overload condition will pass one cheap probe long before it can handle real traffic volume, so requiring sustained evidence of recovery, ideally combined with a slow start ramp, is what prevents the recovery itself from causing the next failure.
Connection draining, also called deregistration delay or graceful shutdown, is missing: instances are being terminated while the load balancer is still routing requests to them, and while requests are still in flight. The correct sequence is to first mark the instance as draining or begin failing its readiness probe, then wait long enough for the load balancer to observe that and stop sending new requests, which takes at least one health check interval times the unhealthy threshold, then stop accepting new connections while allowing in-flight requests to complete up to a bounded timeout, and only then terminate the process. Skipping any of these steps converts an ordinary deploy into a burst of errors that is indistinguishable from an incident on a dashboard.