How ZooKeeper and etcd Use Consensus
On this page
Consensus as a service
Few teams should implement Raft or Paxos. Instead, they use a small, consistent coordination service and build on its primitives. Two dominate: Apache ZooKeeper (from Yahoo, around 2008, used by Hadoop, HBase, older Kafka, Solr) and etcd (from CoreOS, 2013, the store behind every Kubernetes cluster).
Both replicate their data with a consensus protocol across 3 or 5 nodes, so the data survives failures and every client sees one consistent history (CP, Day 5).
ZooKeeper: znodes, watches and ZAB
- Protocol: ZAB (ZooKeeper Atomic Broadcast), a leader-based protocol similar in spirit to Raft.
- Data model: a tree of znodes, like a small file system (
/services/orders/instance-1), each holding a little data. - Ephemeral znodes disappear when the client's session ends, which makes them natural for presence and locks.
- Sequential znodes get an increasing suffix (
lock-0000000042), which gives ordering for fair queues. - Watches: one-time notifications when a znode changes.
- Consistency: writes are linearizable; reads are served by the connected server and can be slightly stale unless the client calls
syncfirst.
etcd: keys, revisions and leases
- Protocol: Raft (Day 16).
- Data model: a flat key-value space with a global revision number that increases with every change; you can read any key at a past revision.
- Leases: attach keys to a lease with a TTL; when the client stops renewing, the lease and its keys expire.
- Watches: streams of changes from a given revision, so clients don't miss events between reconnections.
- Transactions: compare-and-swap style
if/then/elseon key values and revisions. - Consistency: linearizable reads by default (the leader confirms leadership with a quorum), or cheaper serializable reads from any member.
Recipes: service registry, leader election and locks
| Recipe | ZooKeeper | etcd |
|---|---|---|
| Service registry | Ephemeral znode per instance | Key per instance attached to a lease |
| Leader election | Lowest sequential ephemeral znode leads; next watches its predecessor | Campaign with a lease; the election API in client libraries |
| Distributed lock | Sequential ephemeral znodes; fencing via znode version or zxid | Lease plus key creation; fencing via revision |
| Configuration | Znode data with watches | Keys with watches from a revision |
Both expose ordering numbers (zxid, revision) that work as fencing tokens (Day 18). In the ZooKeeper lock recipe, each waiter watches only the znode just before its own rather than the lock as a whole, so a release wakes one client, not all of them at once (the herd effect).
An etcd lock, step by step
Here's the lock recipe with etcdctl, tested against a single etcd 3.6 node. First, a lease that expires after 10 seconds unless the holder renews it:
etcdctl lease grant 10
# lease 694da166a51d5005 granted with TTL(10s)Then a transaction: if the lock key has never been created, create it attached to the lease; otherwise, read who holds it. The input is the condition, a blank line, the operations to run on success, a blank line, and the operations to run on failure:
etcdctl txn <<'EOF'
create("/locks/report") = "0"
put /locks/report worker-a --lease=694da166a51d5005
get /locks/report
EOFworker-a: SUCCESS (the key is created; worker-a holds the lock)
worker-b: FAILURE (the same transaction runs the failure branch instead)
/locks/report
worker-acreate("/locks/report") = "0" means "this key doesn't exist", because a missing key's creation revision is 0. The check and the write happen atomically, so two workers can't both succeed. The key's create_revision (2 here, from etcdctl get /locks/report -w json) is the fencing token: a later holder's key is always created at a higher revision. The holder keeps the lease alive with etcdctl lease keep-alive; if it crashes, the key disappears 10 seconds later and the next worker's transaction succeeds. etcdctl lock and the client libraries' concurrency packages wrap this recipe, and queue waiters in revision order, the same way ZooKeeper uses sequential znodes.
Operating ZooKeeper and etcd
- 3 or 5 nodes, on separate machines and zones. More nodes don't mean more throughput; every write goes through the leader to a majority.
- Fast disks matter: each write is fsynced to the log before acknowledging. Slow disks cause leader elections and timeouts in Kubernetes.
- Keep data small: they're for coordination metadata (kilobytes per key), not application data. etcd defaults to a 2 GB storage quota and periodic compaction of old revisions.
- Back up regularly (etcd
snapshot save), and practice restoring; losing the Kubernetes etcd means losing the cluster's state. - Watch for large numbers of watches or clients reconnecting at once after an incident.
Check yourself
They're deleted automatically when the client's session ends, for example when it crashes or loses its connection for too long. A crashed leader's or service's entry disappears on its own, and watchers are notified to elect a new leader or update their routing.
A global, increasing version for every change, used to read history, resume watches without missing events, and act as a fencing token.
They're designed for small, critical coordination data. Every write goes through consensus and is stored on every node, and etcd keeps the whole keyspace in memory, so large or frequently changing data hurts performance and stability for everyone using the cluster.
It checks that the key doesn't exist yet, since a missing key's creation revision is 0; the key's create_revision, which is higher for every later holder, is the fencing token.
ZooKeeper Atomic Broadcast, the consensus protocol ZooKeeper uses to elect a leader and replicate writes in order to a majority.