Choosing and Operating a Consensus System
On this page
Learning Objective
Combine every decision this course covered - whether consensus is needed at all, fault model, cluster size, algorithm and implementation, coordination service versus embedded, read consistency, and operational guardrails - into one repeatable review sequence, applied end to end to a realistic system design.
Concept Explanation
This is a synthesis lesson rather than new material. The goal is a sequence of questions that can be run against any system that needs distributed agreement, drawing one decision from each earlier lesson, because the transferable skill from this course is the sequence rather than the memory of any individual algorithm detail. Running it takes about twenty minutes on a whiteboard and reliably surfaces the decisions that later become incidents.
The first and most valuable question is whether consensus is needed at all, from the coordination-versus-embedded lesson. A great deal of what gets built on a coordination service can be done with a conditional update against a transactional database that is already deployed, and that option is cheaper to operate, has fewer failure modes, and adds no new availability dependency. Ask specifically what must be agreed, by whom, and what the consequence of disagreement is. If the answer is that one row must not be claimed twice, a conditional update is the right answer. If the answer is that the whole cluster must agree on who the leader is, or on a membership list that survives failures, consensus is genuinely required. Next comes the fault model: crash faults for anything running on machines you own with authenticated peers, Byzantine only when participants are mutually distrusting and have incentive to cheat, with silent corruption handled by checksums rather than by tripling the node count.
Then comes sizing and placement, from the quorum lesson. Three nodes for the common case, five when surviving two simultaneous failures matters, never an even number, and never more than seven. Placement determines whether the partial-synchrony assumption holds, so members belong within a single low-latency network, with non-voting learners or observers used to serve reads elsewhere rather than stretching the voting membership across regions. If a single group cannot carry the write volume, the answer is many groups partitioned across the data, which means embedding rather than a coordination service. Algorithm choice follows almost automatically: use an existing Raft implementation, because the papers-to-production gap documented in "Paxos Made Live" is real, consensus bugs are silent, and no new implementation will match the fault injection and formal verification that etcd's and HashiCorp's libraries have absorbed.
Read consistency is the decision most often left to a default and most often wrong, from the Paxos-versus-Raft lesson. Decide per call site: linearisable reads through ReadIndex where the answer must reflect every committed write, lease reads only if the clock assumption is acceptable and bounded, and explicitly stale reads where staleness is genuinely fine, which is more places than teams assume. Then the client contract: every write idempotent with a deduplication identifier, because a timeout leaves the outcome unknown; every lock classified as efficiency or correctness with fencing tokens validated by the resource in the latter case; and an explicit degraded mode for when coordination is unavailable, chosen from refusing, serving stale within a documented bound, or queueing.
Finally the operational guardrails, which are what actually determine whether the system stays up, from the last two lessons. Alert on healthy member count rather than availability. Alert on leader change rate and on WAL fsync latency. Configure compaction from the first day and schedule defragmentation. Gate every maintenance action on remaining fault tolerance rather than on current health. Add members as learners and promote only after catch-up, one change at a time. Keep verified backups outside the cluster along with the credentials to restore them, ensure the bootstrap path does not depend on the cluster being up, and rehearse the restore. None of these are exotic, and the recurring incidents in this course happened to competent teams that had skipped one of them.
| Decision Point | Question to Ask | This Course's Lesson |
|---|---|---|
| Is consensus needed? | Can a transactional store I already run provide this atomically? | Modules 1, 16 |
| Fault model | Do I own every node, or are participants mutually distrusting? | Module 4 |
| Cluster size and placement | Three or five, odd, and within one low-latency network? | Modules 3, 2 |
| Algorithm and implementation | Which proven Raft library, rather than which paper? | Modules 6, 11 |
| Service or embedded | Is the agreed state small and shared, or the data itself? | Modules 12, 16 |
| Read consistency | Per call site: ReadIndex, lease read, or explicitly stale? | Modules 11, 13, 14 |
| Client contract | Idempotent writes, fenced locks, and a chosen degraded mode? | Modules 15, 18 |
| Operational guardrails | Redundancy alerts, compaction, learner-first resizing, rehearsed restore? | Modules 10, 20 |
Worked Example: Designing Coordination for a Multi-Region Payments Platform
Running the full sequence against one realistic system shows how the questions interact, and in particular how the same platform correctly arrives at three different answers for three different requirements.
# SYSTEM: a payments platform. Three requirements that all sound like
# "we need distributed coordination".
# R1: exactly one instance runs the nightly settlement job
# R2: a payment must never be captured twice
# R3: 200 TB of ledger data, 3 consistent replicas, survive a node loss
# ===== Q1: IS CONSENSUS NEEDED AT ALL? =====
# R1: yes - the whole fleet must agree on one leader. Nothing else
# provides that.
# R2: NO. The payments table is already transactional. Exclusion can be
# enforced by the same transaction that does the work:
UPDATE payments SET status='captured', capture_id=$1
WHERE id=$2 AND status='authorized';
-- 0 rows affected => someone else captured it. No lock, no lease, no
-- pause window, no new availability dependency. Strictly better.
# R3: yes, and it must scale with the data.
# ===== Q2: FAULT MODEL =====
# Our own hardware, our own VPC, mutual TLS between nodes.
# -> CRASH FAULTS. BFT would triple the node count against an adversary
# we do not have. Silent corruption is handled with checksums on log
# entries and ECC memory, not with a BFT protocol.
# ===== Q3: SIZING AND PLACEMENT (for R1) =====
# 5 etcd members, all in region us-east-1, spread across 3 AZs.
# NOT 6 (same tolerance as 5, higher latency, more to fail).
# NOT stretched across regions: a 70ms inter-region RTT against a
# 1000ms election timeout leaves almost no margin, and every commit
# would pay a cross-region round trip.
# eu-west reads are served by LEARNER members: full replication, no
# vote, so the write quorum stays local and fast.
# ===== Q4: ALGORITHM AND IMPLEMENTATION =====
# etcd (Raft). Not a hand-rolled implementation from the paper.
# "Paxos Made Live" documents how much a real system needs beyond the
# algorithm, and consensus bugs are SILENT - they surface only under
# the failure combinations hardest to reproduce.
# ===== Q5: SERVICE OR EMBEDDED =====
# R1: agreed state is one key naming a leader, changing maybe daily.
# -> EXTERNAL COORDINATION SERVICE (etcd).
# R3: agreed state is 200 TB changing constantly, and must scale
# horizontally. One leader cannot carry it.
# -> EMBEDDED CONSENSUS, MANY GROUPS (CockroachDB / TiKV shape:
# one Raft group per range, thousands of groups, unrelated
# writes committing in parallel).
# ===== Q6: READ CONSISTENCY, PER CALL SITE =====
# "who is the settlement leader?" -> ReadIndex (linearisable)
# "what is the fee schedule?" -> stale read + local cache;
# staleness of seconds is fine
# "what is this account's balance?" -> linearisable, always
# No lease reads: they make clock skew a SAFETY dependency, and the
# margin is not worth the saved round trip here.
# ===== Q7: CLIENT CONTRACT =====
# - Every write carries a client-supplied request_id; the server
# deduplicates. A timeout means UNKNOWN, so retries must be safe.
# - R1's leadership is a CORRECTNESS lock -> but rather than fencing
# every downstream write, make the settlement job itself idempotent
# and keyed by settlement date. Cheaper and more robust than fencing.
# - Degraded mode when etcd is unreachable: serve the cached fee
# schedule for up to 5 minutes (counted in a metric), and REFUSE to
# start a settlement run at all. Refusing is correct here: a late
# settlement is recoverable, a double settlement is not.
# ===== Q8: OPERATIONAL GUARDRAILS =====
# ALERTS
# count(up{job="etcd"} == 1) < 5 # LOST REDUNDANCY (page)
# etcd_server_has_leader == 0 # any member
# rate(etcd_server_leader_changes_seen_total) # > 3/day is a problem
# etcd_disk_wal_fsync_duration_seconds p99 > 10ms
# etcd_mvcc_db_total_size_in_bytes > 60% of quota
# CONFIG
# --auto-compaction-retention=1h # from day one
# dedicated local NVMe, never network-attached storage
# scheduled defrag, one member at a time, in a low-traffic window
# PROCEDURE
# maintenance gated on: (healthy - 1) >= quorum, not on "is it green"
# membership changes: learner first, promote after catch-up, ONE
# node at a time, verify health between each
# hourly snapshots to object storage in a DIFFERENT region
# restore credentials stored OUTSIDE etcd (no dependency loop)
# restore rehearsed quarterly in staging, timed and documented
# ===== THE SUMMARY THAT MATTERS =====
# One platform, three requirements, three different correct answers:
# R2 needed no consensus at all - a conditional UPDATE was better.
# R1 needed a small external coordination service.
# R3 needed embedded consensus with many groups.
# Getting this right is almost entirely about asking the questions in
# order, and the first question - "is consensus needed at all?" - is
# the one that saves the most operational cost when the answer is no.Real-World Examples
- CockroachDB demonstrates the multi-answer outcome within a single product: it embeds Raft with one group per data range for the storage layer, because the agreed state is the data and must scale horizontally, while using a much smaller amount of cluster-level coordination for node liveness and settings. The split reflects exactly the reasoning in this lesson - the shape and size of the agreed state, not a general architectural preference, determines whether consensus is embedded or externalised.
- The Kubernetes project applies the same separation in its own operational guidance: it stores all cluster state in a small three or five member etcd cluster kept within one low-latency network, recommends moving the high-churn Event resource to a separate etcd cluster at scale, and documents regular verified snapshots with a restore procedure that does not depend on the control plane being available. Each of those recommendations maps directly to one of the decisions in this review sequence, which is a useful confirmation that the sequence reflects how mature systems are actually operated.
Review Questions & Answers
Because a surprising share of requirements that sound like distributed agreement are satisfied by a conditional update against a transactional store already in production, which costs one round trip, adds no new system to operate, introduces no new availability dependency, and has no pause window because the exclusion is enforced by the same transaction that performs the work. Every alternative - an external coordination service or an embedded consensus layer - adds operational surface, failure modes, and a dependency that must be reasoned about during every incident. Asking the question first means the expensive options are chosen deliberately, for requirements that genuinely need them, rather than by default.
Two things are wrong. Seven members means every commit waits for four acknowledgements rather than three, paying write latency on every request for fault tolerance that will almost never be exercised. Stretching across regions is worse: inter-region round trips of tens of milliseconds sit on the commit path of every write, and the partial-synchrony assumption that Raft's liveness depends on becomes fragile, producing leader churn whenever inter-region latency spikes. The better design is five members within one low-latency network spread across availability zones, with non-voting learners in other regions to serve reads locally, so the write quorum stays fast and local while remote reads are still served nearby.
Per call site, based on what an incorrect answer costs. A read whose answer must reflect every committed write - a current account balance, or which node is the leader before taking an exclusive action - uses a linearisable read through ReadIndex, which costs one round trip and no disk write. A read where staleness of seconds is genuinely harmless, such as a fee schedule or a feature flag, uses an explicitly stale read served from a local cache, which is both faster and more available during a partition. Lease reads are chosen only when the latency saving is genuinely needed and the clock assumption is acceptable and bounded, because they turn clock skew into a safety dependency rather than a liveness one.
Because idempotency makes the overlap window harmless rather than trying to eliminate it, and it requires cooperation from nothing outside the job itself. Fencing tokens are the correct answer when a resource must reject a stale holder, but they require every downstream resource to accept, store, and validate tokens, which is often impossible for third-party APIs and invasive for internal ones. Keying the settlement run by date so that a second execution recognises the work as already done achieves the same safety with a change confined to one component. The general principle from the locking lesson applies: where the operation can be made idempotent, that is usually more robust than making the mutual exclusion perfect.
An alert on healthy member count below the configured membership, which catches the silent loss of redundancy that turns a routine restart into an outage. An alert on leader change rate and on WAL fsync latency percentiles, which catches storage problems that present as unexplained elections. Auto-compaction configured from day one plus scheduled defragmentation, which prevents the backend quota alarm that freezes every write at once. A maintenance gate that checks remaining fault tolerance rather than current health before any node is taken down. A membership procedure that adds learners and promotes only after catch-up, one change at a time. And verified backups stored outside the cluster with restore credentials also held outside it, plus a rehearsed restore, which prevents the dependency loop that turns an incident into a multi-day outage.