Catalog My learning Blog
Raft and Paxos Explained: Consensus, ZooKeeper and etcd · Phase 2 · Lesson 10 · 11 min read

Raft Log Compaction, Snapshots and Membership

On this page
  1. Learning Objective
  2. Concept Explanation
  3. Worked Example: Two Disjoint Majorities From a Membership Change, and the Learner Fix
  4. Real-World Examples
  5. Review Questions & Answers

Learning Objective

Explain why an unbounded replicated log forces snapshotting, describe how a snapshot and the InstallSnapshot path work together, and analyse why naive membership changes can produce two disjoint majorities and how joint consensus and single-server changes prevent it.

Concept Explanation

The replicated log is append-only and therefore grows without limit, which is fine in theory and untenable in practice for two reasons. Storage is the obvious one. The more damaging one is restart time: a node that must replay ten million entries to rebuild its state machine takes minutes or hours to become useful, which converts a routine restart into an outage. Snapshotting solves both. A snapshot is a serialised image of the state machine at a specific log position, recorded together with the last included index and the last included term, after which every log entry up to that index can be discarded. The two metadata fields are not optional bookkeeping: the AppendEntries consistency check needs a prevLogTerm for the entry preceding whatever is being sent, so a node whose log now begins after a snapshot must still be able to answer for the boundary entry.

In the Raft design each server snapshots independently rather than the leader distributing snapshots on a schedule, because every server already holds the same committed entries and can therefore compute the same state on its own. Distributing snapshots would waste bandwidth reproducing something each node can produce locally. The leader only needs to send a snapshot in one circumstance: a follower has fallen so far behind that the entries it needs have already been discarded from the leader's log. That is what the InstallSnapshot RPC is for, and it is a fundamentally different operation from AppendEntries because it transfers state rather than commands, usually in chunks, and the follower discards its own log entirely and rebuilds from the received image. This path is how a newly added member or a long-partitioned node catches up when incremental replay is no longer possible.

Snapshotting has a nasty performance interaction that every implementation has to address. Serialising a large state machine takes time, and blocking the apply path for the duration stalls the entire cluster's progress. The standard solution is copy-on-write: fork the process, or take an immutable reference into a persistent data structure, and serialise that consistent view while normal application continues against the live state. Systems with a storage engine that already supports consistent point-in-time reads, as etcd does through its MVCC backend, get this for free. The other tuning question is frequency, and it is a genuine trade-off: snapshot too rarely and restarts are slow while disk fills with log; snapshot too often and you spend continuous CPU and I/O serialising state that changed very little. etcd exposes this as a snapshot-count setting, defaulting to 100,000 applied entries between snapshots.

Membership changes are the other operation that cannot be done naively, and the reason is worth stating precisely. If every server switched directly from the old configuration to the new one at its own moment of learning about it, there would be an interval during which some servers use the old membership and some use the new one, and during that interval two disjoint majorities can exist. Growing a three-node cluster to five illustrates it: two nodes still using the old configuration constitute a majority of three, while three nodes using the new configuration constitute a majority of five, and those two sets are disjoint. Two leaders could be elected in the same term and both would believe they had a quorum. No network partition is required for this - it is caused entirely by the configuration change itself.

Raft offers two remedies. Joint consensus, described in the original paper, introduces a transitional configuration in which every decision requires separate majorities from both the old and the new membership simultaneously, making disjoint majorities impossible by construction, after which the cluster commits a final entry moving fully to the new configuration. The simpler approach, from Ongaro's dissertation and what etcd and Consul actually implement, is to restrict changes to one server at a time, because adding or removing a single server guarantees the old and new majorities overlap. There is still a trap in the single-server approach: adding a new voting member immediately raises the quorum requirement while that member is still empty and catching up, so a cluster that was tolerating one failure can be pushed to zero remaining tolerance for the duration of the transfer. etcd introduced learner members in version 3.4 specifically for this, letting a new node replicate the full log without counting toward quorum until an operator promotes it once it has caught up.

OperationTriggerRisk If Done NaivelyCorrect Approach
SnapshotApplied entries exceed a configured thresholdBlocking the apply path stalls cluster progressCopy-on-write or an MVCC point-in-time view, serialised in the background
Log compactionA snapshot exists covering entries up to some indexDiscarding entries a lagging follower still needsKeep last included index and term; fall back to InstallSnapshot
Add a memberScaling out or replacing a failed nodeQuorum rises immediately while the new member is emptyAdd as a non-voting learner, let it catch up, then promote
Remove a memberDecommissioning a nodeRemoving the wrong node, or two at once, drops below quorumOne server at a time, verifying cluster health between each change

Worked Example: Two Disjoint Majorities From a Membership Change, and the Learner Fix

The disjoint-majority hazard is entirely arithmetic and needs no network failure to occur, which is what makes it so easy to miss in a design review. The second half shows why etcd learners exist.

Code
# --- THE HAZARD: switching configurations directly ---
# Cluster grows from {A,B,C} to {A,B,C,D,E}.
# Suppose each node switches the instant it learns of the new config.

At some instant:
  A, B    still using C_old = {A,B,C}          -> majority of 3 is 2
  C, D, E already using C_new = {A,B,C,D,E}    -> majority of 5 is 3

  {A,B}     = 2 nodes = a MAJORITY under C_old   -> can elect a leader
  {C,D,E}   = 3 nodes = a MAJORITY under C_new   -> can elect a leader
  These sets are DISJOINT.

# Two leaders, same term, both convinced they hold a quorum, both
# committing. This is genuine split brain, caused purely by the
# configuration change. No network partition was involved at all.


# --- FIX 1: joint consensus (the Raft paper's approach) ---
# Commit a transitional entry C_old,new. While it is in effect, EVERY
# decision needs a majority of C_old AND a majority of C_new:
#   elect/commit requires >=2 of {A,B,C}  AND  >=3 of {A,B,C,D,E}
#   {A,B} satisfies C_old but NOT C_new   -> cannot commit
#   {C,D,E} satisfies C_new but NOT C_old -> cannot commit
# Disjoint majorities are impossible by construction. Once C_old,new is
# committed, commit C_new and the transition completes.


# --- FIX 2: one server at a time (what etcd and Consul do) ---
{A,B,C} -> {A,B,C,D}    # majority 2 -> 3; any old and new majority
{A,B,C,D} -> {A,B,C,D,E} # must overlap when N changes by exactly 1


# --- THE REMAINING TRAP, and why learners exist ---
# Healthy 3-node cluster {A,B,C}: quorum 2, tolerates 1 failure.
# Operator adds empty node D as a VOTING member:
#   membership {A,B,C,D}, quorum is now 3
#   D holds no data and is streaming a 4 GB snapshot for ~10 minutes
#   Effective healthy voters: A, B, C = exactly 3 = exactly quorum
#   -> fault tolerance is now ZERO. If A restarts, the cluster STOPS.
# This is a well-known way to turn a routine scale-out into an outage.

# With a learner (etcd 3.4+, Consul non-voting servers):
etcdctl member add   D --learner --peer-urls=https://d:2380
# D replicates the full log and applies everything, but is NOT counted
# in quorum. Membership stays {A,B,C} + learner D; quorum stays 2;
# fault tolerance stays 1 for the whole catch-up period.

etcdctl endpoint status      # confirm D's raftAppliedIndex has caught up
etcdctl member promote <D-id>   # only NOW does quorum become 3
Log compaction replacing a prefix of entries with a snapshot carrying its last included index and term, and the membership-change hazard in which an old three-node majority and a new five-node majority are disjoint, alongside the learner path that keeps quorum unchanged while a new member catches up.
Log compaction replacing a prefix of entries with a snapshot carrying its last included index and term, and the membership-change hazard in which an old three-node majority and a new five-node majority are disjoint, alongside the learner path that keeps quorum unchanged while a new member catches up.

Real-World Examples

  • etcd introduced learner members in version 3.4 explicitly to address the failure mode where adding a new voting member to a healthy three-node cluster raises the quorum to three while the new member is still streaming a snapshot, leaving zero fault tolerance during the catch-up window. The etcd documentation describes this as one of the most common causes of cluster unavailability during routine maintenance, and the recommended procedure is now to add as a learner, verify the applied index has caught up, and only then promote to voting membership.
  • CockroachDB applies the same pattern at enormous scale, since a cluster contains thousands of independent Raft groups and rebalancing moves replicas between nodes continuously. New replicas join as Raft learners, receive a snapshot of the range, and are promoted to voters only once they are caught up, precisely so that ordinary background rebalancing never transiently reduces the fault tolerance of any range.
WarningAdding a voting member to a consensus cluster raises the quorum requirement immediately, before the new member holds any data, so a healthy three-node cluster momentarily drops to zero fault tolerance while the new node catches up. Always add new members as non-voting learners, confirm they have caught up on applied index, and only then promote. Change membership one server at a time and verify cluster health between each step.

Review Questions & Answers

Because the AppendEntries consistency check requires a node to confirm the term of the entry at prevLogIndex, and after compaction the node no longer holds the entries covered by the snapshot. Retaining the last included index and term lets it still answer for the boundary entry, so a leader sending entries immediately after the snapshot point can have its consistency check satisfied. Without them the node would be unable to validate any incoming entries and would have to be fully re-seeded, and it would also be unable to tell where in the log its reconstructed state actually sits.

Every server has already applied the same committed entries in the same order, so each can compute an identical state image locally. Distributing a snapshot would consume substantial network bandwidth to transfer something the receiver could reproduce from data it already holds. The leader sends a snapshot only in the one case where local computation is impossible: the follower is so far behind that the entries it needs have already been discarded from the leader's log, which is precisely the situation the InstallSnapshot RPC exists to handle.

If servers switch from the old configuration to the new one independently as they learn about the change, there is an interval where the two configurations coexist. Growing three nodes to five makes this concrete: two nodes still on the old configuration form a majority of three, while three nodes already on the new configuration form a majority of five, and those sets share no members. Both groups can therefore elect a leader and commit in the same term, producing genuine split brain. The cause is entirely the overlapping configurations, so a perfectly healthy network provides no protection at all.

Joint consensus introduces a transitional configuration in which every election and every commit requires a majority of the old membership and a majority of the new membership simultaneously. Neither an old-only group nor a new-only group can satisfy both, so disjoint majorities cannot form; once the transitional entry commits, the cluster moves to the new configuration alone. The single-server alternative works because changing the size by exactly one guarantees that any old majority and any new majority must overlap - going from three to four means a majority of two and a majority of three out of overlapping sets that cannot be disjoint - so no transitional state is needed at all. It is simpler to implement and reason about, which is why etcd and Consul chose it.

Adding a fourth voting member raised the quorum from two to three immediately, while the new member held no data and was still streaming a snapshot, so only three members were actually useful and the cluster had exactly quorum with zero remaining fault tolerance. Restarting any one of the three original members dropped the useful count to two against a quorum of three, halting all commits. The correct procedure is to add the node as a learner, which replicates the full log without counting toward quorum, leaving quorum at two and fault tolerance at one throughout the catch-up. Once the learner's applied index has caught up, promoting it to a voter raises quorum to three at a moment when three healthy members genuinely exist.