Catalog My learning Blog
Raft and Paxos Explained: Consensus, ZooKeeper and etcd · Phase 4 · Lesson 20 · 11 min read

Consensus Cluster Failures: Quorum Loss and Recovery

On this page
  1. Learning Objective
  2. Concept Explanation
  3. Worked Example: A Quorum-Loss Incident and the Recovery Decision Tree
  4. Real-World Examples
  5. Review Questions & Answers

Learning Objective

Recognise the recurring operational patterns that take consensus clusters down - quorum loss from routine maintenance, unsafe resizing, storage exhaustion, and dependency loops - and execute a safe recovery procedure when quorum has been lost permanently.

Concept Explanation

Consensus clusters rarely fail because the algorithm is wrong. They fail because of a short list of operational patterns that recur across organisations and across products, and the most valuable thing an engineer can carry into an on-call rotation is recognition of that list. The single most common pattern is quorum loss during routine maintenance, and it almost always has the same shape: a cluster has been running one member short for days or weeks without anyone noticing, because a three-of-five or two-of-three cluster is fully functional in that state and nothing alerts on it. A routine restart then takes out one more member and the cluster drops below quorum instantly. The root cause is monitoring that alerts on unavailability rather than on lost redundancy, which means the degraded state is invisible right up until it becomes an outage.

The second pattern is unsafe resizing, covered mechanically in the membership lesson and worth restating as an operational habit. Adding a voting member raises the quorum immediately, before the new member holds any data, so a healthy three-node cluster momentarily has zero fault tolerance while the newcomer catches up. Removing members in the wrong order, or removing two in quick succession, has the same effect from the other direction. The habit that prevents both is to change membership one node at a time, always add new members as non-voting learners and promote them only after verifying they have caught up, and confirm full cluster health between every step rather than scripting a sequence of changes to run unattended.

The third pattern is storage, and it comes in two forms. Disk exhaustion or quota breach is the etcd case from the Kubernetes lesson: history accumulates without compaction, the backend exceeds its quota, and the cluster goes read-only across every consumer at once. Disk latency is the quieter form: because every Raft commit requires an fsync before acknowledgement, storage slow enough to push fsync latency toward the election timeout produces continuous leader churn that presents as a network problem. Both are prevented by treating the consensus cluster's storage as a first-class concern - dedicated fast local disks, compaction configured from the first day, defragmentation scheduled, and alerting on fsync latency percentiles rather than only on disk space.

The fourth pattern is the dependency loop, and it is the one that turns an incident into a long outage. A coordination service that depends for its own recovery on something that depends on the coordination service cannot be restarted. The classic form is service discovery: nodes find each other through the very service that is down. Another is a Kubernetes cluster whose etcd runs as pods scheduled by the control plane that etcd backs, so recovering etcd requires a working control plane that requires a working etcd. A third is storing the credentials or configuration needed to restore the coordination service inside the coordination service. The prevention is to make bootstrap paths independent of the running system: static peer addresses in configuration, backups and restore credentials held outside the cluster, and a documented, periodically rehearsed procedure that assumes nothing about the system being partially available.

When quorum is permanently lost, recovery follows a fixed order and skipping steps causes the split brain described earlier. First, confirm the loss is genuinely permanent, because a member that will return in an hour is far preferable to any recovery procedure. Second, prefer restoring from a backup over forcing a survivor, because a backup is a known consistent state while a survivor may be missing entries the lost majority had committed. Third, if forcing a survivor is the only option, run the forced reconfiguration on exactly one node, wipe the data directories of every other node completely, and re-add them as fresh members. Fourth, accept and communicate that recently committed writes may have been lost, since a minority survivor cannot know what the majority committed without it. Fifth, verify the recovered cluster is genuinely healthy before restoring traffic. This procedure is worth rehearsing in a non-production environment, because every step is easy to read and easy to get wrong under pressure.

PatternHow It PresentsPrevention
Quorum loss during maintenanceA restart takes down a cluster that was already one member shortAlert on healthy member count, not just on cluster availability
Unsafe resizingFault tolerance drops to zero while a new member catches upAdd as a learner, promote after catch-up, one change at a time
Storage exhaustionCluster goes read-only; every consumer fails writes at onceAuto-compaction from day one, scheduled defrag, alert before the quota
Storage latencyContinuous leader elections that look like a network faultDedicated local SSDs; alert on fsync and commit duration percentiles
Dependency loopThe cluster cannot be restarted because recovery needs itselfStatic bootstrap configuration, external backups, rehearsed restore

Worked Example: A Quorum-Loss Incident and the Recovery Decision Tree

This incident is reconstructed from the shape that recurs most often across organisations, and the recovery decision tree at the end is the part worth rehearsing before it is needed.

Python
# --- THE INCIDENT ---
Day 1   3-node etcd cluster {A,B,C} backing a Kubernetes control plane.
        Node C's host has a failing disk. C is unhealthy.
        Cluster is at 2/3: quorum = 2, still fully functional.
        NO ALERT FIRES. The dashboard is green: writes succeed, the
        API server is responsive, there are no errors anywhere.

Day 9   Routine OS patching. The runbook says "restart nodes one at a
        time, verify the cluster is healthy between each."
        The operator restarts A. The cluster is healthy by every
        metric the runbook checks.
        Healthy members: B only. 1 < quorum of 2.
        -> QUORUM LOST.

Symptoms  kubectl get pods            -> works (served from cache)
          kubectl apply               -> hangs, then times out
          running workloads           -> unaffected, still serving
          new deploys, scaling, HPA   -> all frozen
# The cluster looks "slow", not "down", which delays diagnosis.

Resolution  A finishes rebooting ~4 minutes later, rejoins, quorum is
            restored at 2/3, and everything resumes.
            Total impact: ~4 minutes of a frozen control plane.
            It was luck: had A failed to come back, this would have
            been an unrecoverable 1-of-3 situation.

# --- WHAT WOULD HAVE PREVENTED IT ---
# The runbook checked "is the cluster healthy", which was TRUE at 2/3.
# It needed to check remaining fault tolerance:
def safe_to_restart(members_configured, members_healthy):
    quorum = members_configured // 2 + 1
    return (members_healthy - 1) >= quorum

safe_to_restart(3, 3)   # True  - normal case
safe_to_restart(3, 2)   # False - THE DAY 9 CASE. Should have blocked.

# And the alert that should have fired on Day 1:
#   count(up{job="etcd"} == 1) < 3    for 5m    -> page
# Not: etcd cluster unavailable. By then it is already an outage.


# --- RECOVERY DECISION TREE WHEN QUORUM IS TRULY LOST ---

# Q1: Will any missing member return within a tolerable window?
#     YES -> WAIT. This is by far the best outcome. No data is at risk.
#            Do not start a recovery procedure to save ten minutes.
#     NO  -> continue.

# Q2: Do you have a recent, verified etcd snapshot?
#     YES -> RESTORE FROM BACKUP. Preferred: a snapshot is a known
#            consistent state, whereas a lone survivor may be missing
#            entries the lost majority had already committed.
$ etcdctl snapshot restore backup.db \
      --name n1 --initial-cluster n1=https://n1:2380 \
      --initial-advertise-peer-urls https://n1:2380
#     Then start n1, and add n2/n3 as fresh LEARNERS, promoting each
#     only after it has caught up.
#     NO  -> continue to the last resort.

# Q3: Force a single survivor. LAST RESORT.
#     Choose the survivor with the HIGHEST raft index:
$ etcdctl endpoint status --write-out=table   # compare RAFT INDEX
#     On that node ONLY:
$ etcd --force-new-cluster --data-dir=/var/lib/etcd
#     Then, and this step is NOT optional:
#       - WIPE the data directories of every other node
#       - re-add them as fresh learners, never restart them with old data
#     Restarting another node with its old data, or forcing two nodes,
#     creates genuine split brain - see the split-brain lesson.

# Q4: Communicate the blast radius.
#     A forced survivor cannot know what the lost majority committed
#     without it. Recently acknowledged writes MAY BE GONE. Downstream
#     teams must be told explicitly rather than discovering it later.

# Q5: Verify before restoring traffic.
$ etcdctl endpoint health --cluster
$ etcdctl member list -w table       # exactly the members you expect
$ etcdctl endpoint status -w table   # one leader, indices converging
A timeline of the recurring quorum-loss incident in which an unnoticed unhealthy member turns a routine restart into an outage, paired with the recovery decision tree running from waiting, through restoring a verified snapshot, to forcing a single survivor as a last resort.
A timeline of the recurring quorum-loss incident in which an unnoticed unhealthy member turns a routine restart into an outage, paired with the recovery decision tree running from waiting, through restoring a verified snapshot, to forcing a single survivor as a last resort.

Real-World Examples

  • Roblox's October 2021 outage lasted roughly 73 hours and was traced in its published postmortem to its HashiCorp Consul cluster, where a newly enabled streaming feature interacted with contention in the underlying BoltDB store used for the Raft log, causing write latency to climb until the cluster could no longer function. The postmortem is notable for showing how a consensus-backed coordination layer becomes a total dependency: because service discovery ran through Consul, recovering other systems required Consul to be healthy first, and the team had to restore from a several-hour-old snapshot and disable the problematic feature before the platform could come back.
  • The etcd project documents disaster recovery as a first-class procedure with snapshot save and snapshot restore commands, and explicitly describes forced single-node recovery as unsafe and to be used only when a majority is permanently lost, warning that it can lose recently committed data. The Kubernetes documentation carries the same guidance for control plane backups, recommending regular verified etcd snapshots precisely because restoring from one is strictly preferable to forcing a survivor whose completeness cannot be established.
Best PracticeConsensus clusters are taken down by operations, not by algorithms. Alert on healthy member count rather than on cluster availability, gate every maintenance action on remaining fault tolerance rather than on current health, add members as learners and promote only after catch-up, configure compaction from day one, and keep verified backups plus a bootstrap path that does not depend on the cluster being up. Rehearse the restore procedure before the day you need it.

Review Questions & Answers

Because a cluster running one member short is fully functional - a three-node cluster with two healthy members commits normally and serves every request - so nothing alerts and nobody notices that fault tolerance has silently dropped to zero. The outage occurs later when a routine restart removes one more member. The prevention is to alert on healthy member count against configured membership rather than on cluster availability, so the degraded state is visible while it is still harmless, and to gate maintenance actions on whether the cluster would still hold quorum after the change rather than on whether it is healthy right now.

Because a snapshot is a known consistent state whose contents can be reasoned about, while a survivor's completeness cannot be established. Commitment requires a majority, and a lone survivor may not have been part of the majority that committed the most recent entries, so it may be missing acknowledged writes without any way to detect that. Forcing it makes that incomplete state authoritative and silently discards those writes. A snapshot may be older, but its staleness is a known quantity with a known boundary, which makes the data loss bounded and communicable rather than unknown.

A dependency loop exists when recovering the coordination service requires something that itself depends on the coordination service - nodes discovering each other through the service that is down, etcd running as pods scheduled by the control plane etcd backs, or restore credentials stored inside the cluster being restored. It converts an incident into a long outage because the normal recovery path is unavailable precisely when it is needed, so the team has to improvise a bootstrap under pressure. Prevention means making bootstrap paths independent: static peer addresses in configuration files, backups and credentials held outside the cluster, and a rehearsed procedure that assumes nothing about the system being partially available.

Because a node restarted with its old data directory still holds its previous view of the cluster membership and its own log, so it may assert its own authority rather than accepting state from the forced survivor, and two nodes each considering themselves authoritative is genuine split brain created by the recovery procedure. Wiping the data directory forces the node to join as a fresh member and receive its entire state from the surviving authoritative node, which is the only way to guarantee a single history. This is also why the forced command must be run on exactly one node, since forcing two produces two independent single-node clusters.

Storage latency, specifically the WAL fsync and backend commit duration percentiles. Every Raft commit requires a durable write before acknowledgement, so if fsync latency approaches the election timeout, the leader cannot reliably send heartbeats within it, followers time out, and leadership churns even though every network path is perfect. The replacement leader typically runs on the same class of hardware and hits the same wall. The other candidate causes worth checking are garbage collection pauses and CPU starvation, since both produce the same effect of a leader that is alive but unable to act inside the timeout. Only after those are ruled out is a partial or asymmetric partition worth pursuing.