Network Partitions: What the Minority Side Does
On this page
Learning Objective
Describe what actually happens to clients, applications, and monitoring when a consensus cluster partitions, explain why partial and asymmetric partitions are harder than clean splits, and design application behaviour that degrades sensibly when the minority side stops accepting writes.
Concept Explanation
The textbook statement that the minority side stops accepting writes is accurate and almost useless operationally, because it describes the cluster and says nothing about what anyone connected to it experiences. In practice the effects are uneven in ways that mislead people during incidents. Clients connected to a minority-side node see writes hang until a timeout rather than fail fast, since the node accepts the request and waits for a quorum acknowledgement that never arrives. Reads often keep working, because both ZooKeeper and etcd can serve non-linearisable reads from local state, so a read-heavy dashboard can look entirely healthy while no write has succeeded for ten minutes. And anything using an ephemeral key or lease for liveness will see it expire, so leader elections and membership views change even for applications that were not writing at all.
Clean symmetric partitions, where a set of nodes cannot reach another set and the split is mutual, are the easy case and the one every algorithm is designed against. Real networks produce messier shapes. A partial partition isolates one node from one other node while both remain connected to everyone else, which can produce a leader that a single follower cannot reach: that follower times out and starts elections that disrupt the cluster, while every other node considers the leader perfectly healthy. An asymmetric partition passes traffic in one direction only, so a leader's heartbeats arrive but acknowledgements do not, producing a leader that believes it is isolated and followers that believe it is fine. A gray failure degrades rather than breaks, with packet loss or latency high enough that heartbeats sometimes miss the election timeout, producing intermittent elections that look random. These shapes are why PreVote and CheckQuorum exist and why "the network is fine, I can ping everything" is not a useful diagnostic.
Monitoring for these conditions requires different signals than most teams configure. Alerting on the cluster being completely down catches only the terminal case. The signals that matter earlier are the number of healthy members against the configured membership, which reveals lost redundancy long before lost quorum; the leader change rate, where anything above a handful per day suggests a timing or storage problem; the proposal or commit failure rate, which rises the moment a node cannot reach quorum; and the gap between commit index and applied index, which reveals a state machine falling behind even while consensus is healthy. Tracking healthy member count in particular converts the most common quorum-loss incident, a maintenance restart performed while a member had been quietly down for days, from an outage into a blocked change.
Application behaviour during a partition is a design decision that has to be made in advance, and the default behaviour is almost always wrong. A write to a consensus store that hangs until timeout leaves the caller in the genuinely worst state: the outcome is unknown, because the request may have been committed just as the response was lost. Every write must therefore be idempotent, carrying a client-supplied request identifier the system can deduplicate against, so a retry after a timeout is safe. Beyond that, the application needs an explicit answer for what it does when coordination is unavailable, and there are only a few honest choices: refuse the operation, proceed using the last known cached coordination state with a documented staleness bound and accept the risk, or queue the work for later. Choosing none of these means choosing whichever one the timeout behaviour happens to produce.
There is one more property worth internalising, because it is frequently mistaken for a bug. A consensus cluster that loses quorum does not degrade gracefully into a slower or weaker mode; it stops. This is the deliberate consequence of choosing consistency over availability, and it means that recovery from a partition is a step function rather than a ramp. When the partition heals, the minority side catches up by log replication or a snapshot transfer, the previously isolated nodes rejoin, and the cluster resumes at full speed, but nothing useful happens in between. Systems that need to keep serving during a partition must either not use consensus for that path, by caching or by accepting eventual consistency, or must be partitioned so that most requests only need a quorum that is local to one site.
| Partition Shape | What It Looks Like | Typical Consequence |
|---|---|---|
| Clean symmetric split | Two sets mutually unreachable | Majority side continues; minority stalls. The designed-for case |
| Partial partition | One node cannot reach one other; all others fine | The isolated pair disrupts elections while the cluster looks healthy |
| Asymmetric partition | Traffic flows one direction only | Leader believes it is isolated while followers believe it is fine |
| Gray failure | High packet loss or latency, not a clean break | Intermittent elections that appear random and resist diagnosis |
| Quorum loss | No side reaches a majority | Everything stops: no leader, no commits, reads may still succeed |
Worked Example: What Each Participant Observes During a Partition
The reason partitions are confusing during an incident is that every observer has a different and internally consistent picture. Laying them side by side is the fastest way to learn to recognise the pattern.
# 5-node etcd cluster across two racks. Rack1 = {A(leader), B}.
# Rack2 = {C, D, E}. The link between racks fails at t=0.
# ---- WHAT THE APPLICATION ON RACK 1 SEES ----
t=0.0 PUT /config/x -> hangs
t=5.0 context deadline exceeded
# The outcome is UNKNOWN, not failed. The request may have been
# committed just as the response was lost. This is why every write
# needs a client-supplied request id the server can deduplicate.
GET /config/x -> returns a value instantly (serializable read from
local state). The app may conclude "etcd is fine".
# ---- WHAT THE APPLICATION ON RACK 2 SEES ----
t=0.0 PUT /config/x -> hangs (no leader yet)
t=1.1 new leader elected on rack 2
t=1.2 PUT /config/x -> succeeds normally
# From here everything works. This side has no idea anything is wrong
# beyond two members showing as unhealthy.
# ---- WHAT A LEADER-ELECTION CLIENT ON RACK 1 SEES ----
t=10 its lease expires (keep-alives cannot reach a quorum)
-> it MUST stop acting as leader immediately
# Note it was not writing anything. Liveness state expires regardless.
# ---- WHAT MONITORING SEES, IF ONLY THE WRONG THINGS ARE WATCHED ----
etcd_server_has_leader{instance="C"} 1 # rack 2: fine
etcd_server_has_leader{instance="A"} 0 # rack 1: the real signal
read QPS: unchanged # reads still served locally
error rate on the dashboard: near zero # writes HANG, they do not
# return errors
# A dashboard built on error rate and read QPS shows a healthy system.
# ---- SIGNALS THAT ACTUALLY CATCH IT ----
etcd_server_has_leader == 0 # any member
etcd_server_leader_changes_seen_total rate # > a few per day is bad
etcd_server_proposals_failed_total rate # rises immediately
count(up{job="etcd"} == 1) < <configured members> # LOST REDUNDANCY,
# the single most valuable alert: it fires when the cluster is still
# fully working but its fault tolerance has silently dropped.
# ---- DESIGNING THE APPLICATION FOR THIS ----
def get_config(key):
try:
v = etcd.get(key, timeout=2.0)
cache.set(key, v, fetched_at=now())
return v
except (Timeout, NoQuorum):
cached = cache.get(key)
if cached and now() - cached.fetched_at < MAX_STALENESS:
metrics.increment('config.served_stale')
return cached.value # DOCUMENTED degraded mode
raise ConfigUnavailable # deliberate refusal, not a hang
def write_config(key, value, request_id):
# Idempotent by construction: a retry after a timeout is safe
# because the server deduplicates on request_id.
return etcd.txn(
compare=[Compare(f'/requests/{request_id}', 'create_revision', 0)],
success=[Put(key, value), Put(f'/requests/{request_id}', '1', lease=day_lease)],
failure=[], # already applied: nothing to do
)
# The three honest options when coordination is unavailable are: refuse,
# serve stale within a documented bound, or queue for later. Choosing
# none of them means choosing whatever the default timeout produces.Real-World Examples
- Kubernetes control planes commonly exhibit the misleading-symptom pattern during an etcd partition: kubectl reads continue to work because the API server can serve from its watch cache, while every create, update, and delete hangs until timeout, so operators initially report the cluster as slow rather than as down. The etcd documentation's recommended alerts are built around leader presence, leader change rate, and failed proposals for exactly this reason, since none of those symptoms are visible in application error rates.
- Jepsen, Kyle Kingsbury's distributed systems testing project, has repeatedly demonstrated that partial and asymmetric partitions uncover failures that clean symmetric splits do not, across analyses of etcd, Consul, ZooKeeper, MongoDB, Elasticsearch, and many others. The finding that recurs is that systems handle the clean case they were designed and tested against, while one-way connectivity and single-node isolation produce election storms and stale reads that only appear under those specific shapes.
Review Questions & Answers
Because reads continue to be served from local node state without needing a quorum, so read throughput is unchanged, and writes hang until a client timeout rather than returning errors, so server-side error rates stay near zero. A dashboard built on error rate and request volume therefore shows normal operation while nothing is being committed. The revealing signals are leader presence per member, leader change rate, the failed proposal rate, and above all the count of healthy members against configured membership, which is the only one that fires while the cluster is still fully functional but has lost its redundancy.
Because the two nodes disagree with everyone else rather than splitting the cluster into two consistent views. If the isolated pair happens to be a follower and the leader, that follower times out, becomes a candidate, and disrupts a cluster that every other member considers perfectly healthy, producing elections with no apparent cause. Ordinary diagnostics mislead, since most node pairs can reach each other and any spot check of connectivity looks fine. This is precisely the shape PreVote mitigates, by preventing a node that cannot win from raising the term, and it is why Jepsen-style testing deliberately includes partial and asymmetric partitions rather than only clean splits.
The request may have been received, replicated to a majority, and committed at the exact moment the response was lost or the partition formed, so a timeout distinguishes nothing between "never applied" and "applied, acknowledgement lost". Treating it as a failure and retrying blindly risks applying the operation twice. The application must therefore make writes idempotent, attaching a client-supplied request identifier that the server records and deduplicates against, so a retry after a timeout either applies the operation once or recognises it as already applied. Any design that retries non-idempotent writes after a timeout will eventually produce duplicates.
The options are to refuse the operation with a clear error, to proceed on the last known cached coordination state within a documented staleness bound while recording that it happened, or to queue the work for when coordination returns. Doing nothing is not a fourth option because the application still behaves somehow: it inherits whatever the client library's default timeout produces, usually threads blocking for seconds and then throwing, which exhausts connection pools and turns a coordination outage into a full application outage. Choosing deliberately converts an unbounded failure into a bounded one, and the choice differs per call site, since serving a stale configuration value is often fine while proceeding without a leadership check may not be.
Because the algorithm is built to sacrifice liveness rather than safety: with no majority reachable there is no way to commit without risking divergence against a possible majority elsewhere, so it commits nothing. There is no weaker mode to fall back to, since any such mode would mean accepting writes without quorum, which is exactly the split brain the design exists to prevent. Recovery is therefore a step function rather than a ramp. Systems that must keep serving during a partition have to take that path off consensus entirely, by caching with a documented staleness bound or by accepting eventual consistency for that data, or must partition the workload so that most requests need only a quorum local to one site.