Split Brain Explained: How Quorum Prevents It
On this page
Learning Objective
Define split brain precisely, explain why quorum makes it arithmetically impossible within a single consensus group, and identify the boundary cases where a consensus-backed system can still produce divergence - configuration errors, a consensus layer controlling a non-consensus data layer, and forced recovery operations.
Concept Explanation
Split brain is the state in which two or more parts of a system each believe they are the authoritative owner of some resource and each independently accepts and commits writes, producing durable divergent state that cannot be reconciled without discarding acknowledged data. The emphasis on committing matters. Two nodes each believing they are leader is not split brain and is in fact routine during any failover; what makes it split brain is that both are able to make their beliefs durable. Everything in this course has been building toward the mechanism that prevents this, and it is worth stating in its final form: a write commits only with acknowledgement from a majority, two majorities of the same set cannot be disjoint, and therefore at most one side of any division can commit. Not "is unlikely to", but cannot.
Because the guarantee is arithmetic, split brain inside a correctly configured single consensus group is not a risk to be mitigated but an impossibility. The practical consequence is that when a team reports split brain in a system backed by Raft or Paxos, the cause is essentially never the consensus algorithm. It is always one of a small number of boundary conditions, and knowing that list turns an open-ended investigation into a checklist.
The first and most common boundary is a consensus layer making decisions about a data layer that is not itself consensus-backed. A failover tool that runs Raft internally can hold a perfectly consistent, majority-agreed opinion about which database should be primary, and still cause divergence, because the databases underneath are replicating asynchronously and the promoted replica may be missing writes the old primary had already acknowledged. The consensus was correct; it was simply about the wrong thing. GitHub's 2018 outage is the canonical example of exactly this shape, and the lesson generalises: agreement about who is the leader does not imply agreement about what the leader has.
The second boundary is configuration, and it has historically been the most damaging in practice because it produces a system that looks like a quorum system and is not. Any deployment where the quorum size is set manually, or where nodes disagree about the membership list, or where two independent groups are accidentally configured against the same data, can produce two genuine majorities of two different sets. Elasticsearch before version 7.0 is the canonical illustration: operators had to manually set a minimum master nodes value to a majority of the master-eligible nodes, and getting it wrong produced real, documented split brain with divergent indices. The fix in version 7.0 was to compute the quorum automatically from a cluster-managed voting configuration rather than trusting an operator-supplied number, which is a useful design principle in its own right: safety-critical arithmetic should not be a configuration field.
The third boundary is deliberate and comes with a warning label. Forced recovery operations, such as etcd's force-new-cluster flag or an equivalent unsafe reconfiguration, exist so that an operator can restore service from a minority when the majority is permanently lost. They work by telling a surviving node to consider itself the whole cluster, which is precisely a manual override of the quorum rule, and if any of the supposedly lost nodes returns, or if two survivors are forced independently, the result is genuine split brain created by the recovery procedure. This is why every such command is documented as a last resort, why it should be run on exactly one node, and why the other nodes must be wiped rather than restarted. A fourth, rarer boundary is worth knowing: two consensus groups that are each internally consistent but that both control the same external resource, such as two independent clusters writing to the same object store bucket, where no single quorum exists to arbitrate at all.
| Situation | Is It Split Brain? | Why |
|---|---|---|
| Two nodes both believe they are leader during a partition | No | Only the majority side can commit; the other holds a title with no quorum |
| A minority-side leader accepts writes it never acknowledges | No | Nothing was committed or promised, so the uncommitted tail is safely truncated |
| Consensus failover promoting an async replica missing writes | Yes, effectively | The consensus decision was correct but the data layer had no quorum guarantee |
| Manually configured quorum size set too low on some nodes | Yes | Two different membership views can each produce a genuine majority |
| Two operators independently forcing single-node recovery | Yes | The quorum rule was deliberately overridden on two nodes at once |
Worked Example: Why the Arithmetic Holds, and the Three Ways Around It
Seeing the impossibility proof next to the exceptions makes clear that the exceptions are not weaknesses in the algorithm but places where the algorithm is not actually in the loop.
# --- WHY SPLIT BRAIN IS IMPOSSIBLE INSIDE ONE CONSENSUS GROUP ---
# 5 nodes. Quorum = 3. Any partition splits them into groups summing to 5.
#
# 5 | 0 -> one side has 5 >= 3, the other has 0. One committer.
# 4 | 1 -> 4 >= 3, 1 < 3. One committer.
# 3 | 2 -> 3 >= 3, 2 < 3. One committer.
# 2 | 2 | 1 -> none reaches 3. ZERO committers.
#
# There is no split of 5 nodes into two groups that BOTH reach 3,
# because 3 + 3 = 6 > 5. This is not a probability argument. There is
# no network failure, no timing, and no adversary that produces two
# committing sides. The worst case is zero committers: unavailability.
# --- EXCEPTION 1: consensus about the wrong thing (GitHub 2018 shape) ---
# Orchestrator runs Raft and correctly, unanimously decides:
# "East primary is unreachable -> promote West replica."
# But the MySQL replication underneath is ASYNCHRONOUS:
# East primary had acknowledged writes up to position 5000
# West replica had received only up to position 4850
# After promotion:
# West accepts NEW writes starting from 4850
# East (still alive, just unreachable) holds 4851..5000, acknowledged
# Two divergent histories of acknowledged writes.
#
# The consensus layer was flawless. It agreed about LEADERSHIP.
# It could not agree about DATA, because the data layer had no quorum.
# Rule: a consensus-backed failover controller in front of an async
# replication layer gives you consistent decisions about an inconsistent
# system.
# --- EXCEPTION 2: the quorum number is a config field ---
# Elasticsearch before 7.0: discovery.zen.minimum_master_nodes
# 5 master-eligible nodes; correct value is 3.
# nodes A,B,C configured with minimum_master_nodes: 3 (correct)
# nodes D,E configured with minimum_master_nodes: 2 (typo)
# Partition {A,B,C} | {D,E}:
# {A,B,C} = 3 >= 3 -> elects a master. Correct.
# {D,E} = 2 >= 2 -> ALSO elects a master. Both sides accept indexing.
# Two masters, two divergent indices, real data loss on rejoin.
#
# Fixed in 7.0 by computing the voting configuration inside the cluster
# rather than trusting an operator-supplied number. Design principle:
# never make safety-critical arithmetic a configuration field.
# --- EXCEPTION 3: deliberately overriding quorum during recovery ---
# 3-node etcd cluster; nodes B and C are believed permanently lost.
$ etcd --force-new-cluster # run on A ONLY
# A now considers itself a complete 1-node cluster: quorum = 1.
# This is a MANUAL OVERRIDE of the quorum rule, and it is the only way
# back from permanent quorum loss.
#
# It creates split brain if EITHER of these happens:
# - B or C is later restarted with its old data instead of wiped
# - a second operator runs the same command on B "to be safe"
# Then two single-node clusters each believe they are authoritative.
#
# Correct procedure: force on exactly ONE node, WIPE the data
# directories of all others, and re-add them as fresh learners.Real-World Examples
- GitHub's October 2018 incident began with a 43-second connectivity loss between its US East Coast network hub and its primary US East Coast data centre. Orchestrator, which uses Raft for its own coordination, correctly promoted a US West Coast MySQL cluster to primary. Because MySQL replication between the coasts was asynchronous, the East Coast cluster held writes that had been acknowledged to users but never reached the West Coast, while the West Coast then accepted new writes. Restoring consistency required more than 24 hours of degraded service, and GitHub's public postmortem is unusually clear that the coordination layer behaved correctly and the divergence came from the replication layer beneath it.
- Elasticsearch before version 7.0 required operators to manually set discovery.zen.minimum_master_nodes to a majority of master-eligible nodes, and misconfiguring it was a documented and frequently reported cause of split brain producing divergent indices. Version 7.0 replaced the setting with an automatically maintained voting configuration computed by the cluster itself, an explicit acknowledgement that leaving the quorum calculation to operator configuration had been a design error rather than a flexibility feature.
Review Questions & Answers
Because belief without quorum is inert. A node isolated on the minority side can hold the title of leader, append to its local log, and accept client connections, but it cannot reach a majority, so it cannot advance its commit index, cannot acknowledge anything to a client, and cannot make any of it durable. Split brain requires two sides both committing, producing divergent acknowledged state. Since two majorities of the same set cannot exist simultaneously, at most one side can commit, so simultaneous belief in leadership is a routine and harmless condition during any failover.
The consensus was about leadership, not about data. Orchestrator correctly and unanimously decided to promote a West Coast MySQL cluster after the East Coast became unreachable, but MySQL replication between the coasts was asynchronous, so the promoted replica was missing writes the East Coast primary had already acknowledged to users. Once the West Coast began accepting new writes, two divergent histories of acknowledged data existed. The general lesson is that a consensus-backed controller in front of a non-consensus data layer gives you consistent decisions about an inconsistent system: agreement on who leads does not imply agreement on what the leader holds.
Because the quorum calculation is the entire safety mechanism, and exposing it as a configuration field made a single typo sufficient to break it silently. With different nodes holding different minimum-master values, a partition could leave two groups each satisfying its own threshold, producing two genuine masters and divergent indices with no warning until reconciliation. The value is also easy to get wrong when a cluster is resized, since it must be updated in step. Version 7.0 moved the calculation inside the cluster as an automatically maintained voting configuration, following the principle that safety-critical arithmetic should be derived by the system rather than supplied by a human.
Because it works by instructing a surviving node to consider itself the entire cluster, which is a deliberate manual override of the quorum rule: the node now commits alone. If one of the supposedly lost nodes is later restarted with its old data, or if a second operator runs the same command on another survivor, two nodes each consider themselves authoritative and both accept writes, which is textbook split brain created by the recovery itself. The containing procedure is to run the command on exactly one node, wipe the data directories of every other node rather than restarting them, and re-add them as fresh members so they receive state from the forced survivor instead of asserting their own.
No. Quorum guarantees agreement within a single consensus group about that group's own log; it says nothing about coordination between two groups that share an external resource. Each cluster reaches a perfectly valid majority decision about its own state, and both then write to the shared bucket with no arbiter between them, so the bucket ends up with interleaved or overwritten data that neither cluster knows about. This reveals that the guarantee is scoped to a group, not to a resource: protecting a shared resource requires either a single consensus group owning it or fencing tokens the resource itself validates, which is the same conclusion the locking lesson reached from a different direction.