Catalog My learning Blog
Raft and Paxos Explained: Consensus, ZooKeeper and etcd · Phase 2 · Lesson 9 · 11 min read

Raft Failure Handling: Partitions and Leader Loss

On this page
  1. Learning Objective
  2. Concept Explanation
  3. Worked Example: A Minority-Side Leader, and a Term-Inflating Rejoin
  4. Real-World Examples
  5. Review Questions & Answers

Learning Objective

Predict what a Raft cluster does when the leader is isolated on the minority side of a partition, explain the disruptive-server problem caused by a partitioned node inflating its term, and describe the PreVote and CheckQuorum extensions that production implementations add to address both.

Concept Explanation

The most instructive Raft failure is a network partition that leaves the leader on the minority side, because every important property of the algorithm shows itself at once. Consider a five-node cluster with leader A partitioned together with B, while C, D, and E remain connected to each other. A continues to believe it is leader and continues to accept client writes, appending them to its local log. It cannot replicate them to a majority, so it can never advance its commit index, so it never acknowledges any of those writes to clients. Meanwhile C, D, and E stop hearing heartbeats, hold an election, and one of them wins with three votes out of five, because three is a majority. The cluster now has one node that thinks it is leader for term 7 and one that genuinely is leader for term 8, and the crucial point is that this is not split brain: only one of them can commit anything, because only one side holds a majority.

When the partition heals, resolution is mechanical rather than negotiated. A receives a message carrying term 8, which exceeds its own term 7, and the unconditional rule from the term mechanism makes it revert to follower immediately. The new leader then runs the consistency check against A's log, finds the last matching index, and truncates everything A appended during the partition. Those entries are discarded permanently, and no correctness rule is violated by discarding them, because they were never committed and therefore never acknowledged to any client. This is why the client-facing contract matters so much: a write is only durable once the leader has told the client it succeeded, and during the partition A told no one anything. In practice a well-behaved client library sees requests to A time out rather than succeed.

That leaves a real operational nuisance. A minority-side leader keeps accepting connections and sitting on client requests until they time out, which wastes time and can confuse load balancers. The CheckQuorum extension addresses this directly: a leader periodically verifies it has received responses from a majority within an election timeout, and steps down voluntarily if it has not. With CheckQuorum enabled, the isolated leader in the example above demotes itself within roughly one election timeout, so clients get a fast redirect or failure rather than hanging.

The second problem is subtler and bites real deployments regularly. A node partitioned away from the cluster times out, increments its term, becomes a candidate, gets no votes, times out again, increments again, and repeats. After five minutes of isolation its term may be thousands ahead of the cluster's. When the partition heals, its very first RequestVote carries that enormous term, and every healthy node including the current leader must adopt the higher term and revert to follower under the unconditional term rule. The rejoining node is too far behind on its log to win the resulting election, so a perfectly functional cluster is forced into a needless leadership change by a node that cannot even become leader. The PreVote extension, described in Ongaro's dissertation and implemented in etcd and HashiCorp's raft library, fixes this: before incrementing its term, a candidate runs a preliminary round asking peers whether they would grant it a vote given its current log and term. Peers refuse if they have heard from a leader recently. Only if the pre-vote succeeds does the node increment its term for real, so an isolated node never inflates its term at all.

Two further behaviours are worth knowing because they surprise operators. First, a Raft cluster that loses quorum does not fail loudly in a way most monitoring catches by default: reads served from local state may keep succeeding while writes hang, so a dashboard tracking only error rates can look healthy while the cluster commits nothing. Second, recovery after quorum loss is not automatic if the lost nodes are permanently gone. A three-node cluster that loses two nodes irrecoverably cannot elect a leader, and the only path back is an unsafe forced reconfiguration - etcd's force-new-cluster flag or an equivalent - which rewrites the membership on the survivor and explicitly risks losing recently committed entries. That is a deliberate last resort, documented as such, and it is the reason backups and careful membership management matter more in consensus systems than in most databases.

Failure ScenarioMajority SideMinority SideResolution on Heal
Leader isolated with a minorityElects a new leader, keeps committingOld leader accepts writes it can never commit or acknowledgeOld leader sees a higher term, steps down, uncommitted tail truncated
Leader crashes outrightElection after one election timeout, service resumesNot applicableRestarted node rejoins as a follower and catches up
Node partitioned alone for minutesUnaffected while isolatedTerm inflates on every failed electionWithout PreVote, forces a needless election on rejoin; with PreVote, no effect
Quorum lost entirelyNone existsEverything stalls: no leader, no commitsOnly restoring a member recovers it safely; forced reconfiguration risks data loss

Worked Example: A Minority-Side Leader, and a Term-Inflating Rejoin

These two failure modes are frequently confused with each other and with split brain, so tracing them side by side clarifies which one costs data, which one costs only availability, and which one costs nothing but an unnecessary election.

Code
# 5 nodes: A B C D E. A is leader in term 7. Majority = 3.
# A network partition splits {A, B} from {C, D, E}.

# --- MINORITY SIDE ---
t=0     A accepts client write "x=1", appends at index 50.
        A -> B : AppendEntries(index=50)   -> B acknowledges.
        matchIndex: {A:50, B:50, C:49, D:49, E:49}
        Majority at index 50? Only 2 of 5. NO.
        commitIndex stays at 49. The client is NOT told anything.
t=1s    A has accepted 400 more writes. commitIndex still 49.
        Every one of those client requests eventually TIMES OUT.
        With CheckQuorum enabled, A notices it has had no majority
        response within an election timeout and STEPS DOWN at ~t=1s,
        so clients fail fast instead of hanging.

# --- MAJORITY SIDE ---
t=0.3s  C, D, E stop hearing heartbeats. D's timeout fires first.
        D -> C,E : RequestVote(term=8, lastLogIndex=49, lastLogTerm=7)
        C, E grant. D has 3 of 5 = MAJORITY -> leader for term 8.
        D appends a no-op at index 50 (term 8), commits it,
        and continues serving clients normally.

# NOT SPLIT BRAIN: A has a title but no quorum, so it commits nothing.
# Exactly one side can ever hold a majority, so exactly one side commits.

# --- PARTITION HEALS at t=60s ---
A -> C : AppendEntries(term=7, ...)
C -> A : {term: 8, success: false}
A adopts term 8 and reverts to follower.
D (leader) runs the consistency check against A:
   A's index 50 has term 7; D's index 50 has term 8 -> conflict.
   A truncates indices 50..450 and takes D's entries instead.
# The 401 writes A accepted during the partition are gone. No rule is
# violated: none was ever committed, so none was ever acknowledged.


# --- THE DISRUPTIVE REJOIN (a different, sneakier problem) ---
# Node E is partitioned ALONE for 5 minutes. Election timeout ~300ms.
E: term 8 -> 9 -> 10 -> ... -> 1008    # one bump per failed election
# E cannot win (no peers reachable) but nothing stops it counting up.

# Partition heals. E's first act:
E -> A,B,C,D : RequestVote(term=1008, lastLogIndex=49, ...)
# EVERY node, including the healthy leader, must adopt term 1008 and
# revert to follower - the higher-term rule is unconditional.
# E then LOSES the election anyway: its log is 400 entries behind, so
# the up-to-date check denies it every vote.
# Net effect: a node that could never be leader forced a healthy cluster
# through an unnecessary leadership change and a service gap.

# --- WITH PREVOTE ENABLED ---
E -> A,B,C,D : PreVote(term_would_be=9, lastLogIndex=49, lastLogTerm=7)
A,B,C,D      : "I heard from a leader within the last election timeout,
                and your log is behind mine." -> DENIED
# E never increments its real term at all. It stays at term 8 for the
# entire five minutes, rejoins silently as a follower, and catches up.
# The healthy cluster never notices.
A five-node cluster split three to two: the majority side elects a new leader and keeps committing while the isolated old leader accepts writes it can never commit or acknowledge, plus the term-inflation problem of a long-isolated node and how a pre-vote round prevents it from disrupting the cluster on rejoin.
A five-node cluster split three to two: the majority side elects a new leader and keeps committing while the isolated old leader accepts writes it can never commit or acknowledge, plus the term-inflation problem of a long-isolated node and how a pre-vote round prevents it from disrupting the cluster on rejoin.

Real-World Examples

  • etcd implements both PreVote and CheckQuorum and recommends enabling PreVote, documenting the exact disruptive-server scenario: a member isolated by a network partition inflates its term through repeated failed elections and, on rejoining, forces the healthy leader to step down despite being unable to win. Kubernetes clusters with flaky inter-node networking see this as unexplained control plane leader changes correlated with a node recovering rather than with a node failing.
  • HashiCorp's raft library, used by Consul, Nomad, and Vault, added PreVote support and documents the same failure mode, alongside guidance that operators seeing repeated leadership elections should check for network instability affecting a single member rather than assuming a problem with the leader itself. The diagnostic signature is characteristic: leadership changes that begin at the moment a previously unreachable member becomes reachable again.
Best PracticeA leader isolated on the minority side is not split brain - it holds a title with no quorum, so it commits nothing and acknowledges nothing, and its uncommitted tail is safely truncated when the partition heals. The genuinely disruptive failure is subtler: a node isolated for minutes inflates its term through failed elections and forces a healthy cluster into a needless leadership change on rejoin. Enable PreVote to stop the inflation and CheckQuorum so an isolated leader demotes itself rather than leaving client requests to hang.

Review Questions & Answers

The writes are discarded when the partition heals, and it is not a correctness violation. The isolated leader could only replicate to two of five nodes, never a majority, so it could never advance its commit index and therefore never acknowledged any of those writes to a client. Raft's durability contract is that an acknowledged write survives, and nothing here was acknowledged; clients saw timeouts. On healing, the old leader sees the higher term, steps down, and the new leader truncates the uncommitted tail. The useful lesson for application authors is that a request timeout means the outcome is genuinely unknown and must be retried idempotently, not that the write definitely failed.

Split brain means two sides both accepting and committing conflicting writes, producing divergent durable state. Here only the majority side can assemble a quorum, so only it can advance a commit index or acknowledge anything. The minority-side leader holds nothing more than a stale title: it can append to its own local log and nothing else. Because two majorities of the same cluster cannot exist simultaneously, at most one side can ever commit, which is the arithmetic guarantee that makes believing yourself leader harmless when you cannot prove it to a quorum.

An isolated node times out repeatedly and increments its term on each failed election, so after minutes of isolation its term far exceeds the cluster's. On rejoining, its RequestVote carries that inflated term, and the unconditional higher-term rule forces every node including the healthy leader to step down, even though the rejoining node cannot win because its log is behind. PreVote adds a preliminary round in which the candidate asks whether peers would vote for it, without incrementing its term; peers refuse if they have heard from a leader recently. Safety is untouched because the pre-vote round grants nothing and changes no state - it is purely a filter that prevents a doomed candidacy from raising the term in the first place.

CheckQuorum makes a leader periodically verify that it has received responses from a majority of the cluster within an election timeout, and step down voluntarily if it has not. Without it, a leader isolated on the minority side keeps its title indefinitely, accepting client connections and holding requests until they individually time out, which leaves clients hanging and can keep the node in a load balancer pool. With it, the isolated leader demotes itself within roughly one election timeout, so clients get a fast failure or redirect toward the side that can actually commit.

The single survivor cannot form a majority of three, so it cannot elect a leader or commit anything, and no amount of waiting changes that. A forced reconfiguration rewrites the membership so the survivor considers itself a single-node cluster and can commit alone. The risk is concrete: the survivor may be missing entries that were committed on the two lost nodes, since commitment required only two of three and the survivor may not have been in that pair. Forcing it to become authoritative silently discards those entries, so the operation is documented as a last resort to be used only when the lost nodes are genuinely unrecoverable and losing recent writes is preferable to remaining down.