Catalog My learning Blog
Distributed Systems Explained: CAP, Clocks and Replication · Phase 3 · Lesson 19 · 5 min read

How ZooKeeper and etcd Use Consensus

On this page
  1. Consensus as a service
  2. ZooKeeper: znodes, watches and ZAB
  3. etcd: keys, revisions and leases
  4. Recipes: service registry, leader election and locks
  5. An etcd lock, step by step
  6. Operating ZooKeeper and etcd
  7. Check yourself

Consensus as a service

Few teams should implement Raft or Paxos. Instead, they use a small, consistent coordination service and build on its primitives. Two dominate: Apache ZooKeeper (from Yahoo, around 2008, used by Hadoop, HBase, older Kafka, Solr) and etcd (from CoreOS, 2013, the store behind every Kubernetes cluster).

Both replicate their data with a consensus protocol across 3 or 5 nodes, so the data survives failures and every client sees one consistent history (CP, Day 5).

ZooKeeper offers a tree of znodes with watches and ephemeral nodes over ZAB; etcd offers a flat key space with revisions, leases and watches over Raft.
ZooKeeper offers a tree of znodes with watches and ephemeral nodes over ZAB; etcd offers a flat key space with revisions, leases and watches over Raft.

ZooKeeper: znodes, watches and ZAB

  • Protocol: ZAB (ZooKeeper Atomic Broadcast), a leader-based protocol similar in spirit to Raft.
  • Data model: a tree of znodes, like a small file system (/services/orders/instance-1), each holding a little data.
  • Ephemeral znodes disappear when the client's session ends, which makes them natural for presence and locks.
  • Sequential znodes get an increasing suffix (lock-0000000042), which gives ordering for fair queues.
  • Watches: one-time notifications when a znode changes.
  • Consistency: writes are linearizable; reads are served by the connected server and can be slightly stale unless the client calls sync first.

etcd: keys, revisions and leases

  • Protocol: Raft (Day 16).
  • Data model: a flat key-value space with a global revision number that increases with every change; you can read any key at a past revision.
  • Leases: attach keys to a lease with a TTL; when the client stops renewing, the lease and its keys expire.
  • Watches: streams of changes from a given revision, so clients don't miss events between reconnections.
  • Transactions: compare-and-swap style if/then/else on key values and revisions.
  • Consistency: linearizable reads by default (the leader confirms leadership with a quorum), or cheaper serializable reads from any member.

Recipes: service registry, leader election and locks

RecipeZooKeeperetcd
Service registryEphemeral znode per instanceKey per instance attached to a lease
Leader electionLowest sequential ephemeral znode leads; next watches its predecessorCampaign with a lease; the election API in client libraries
Distributed lockSequential ephemeral znodes; fencing via znode version or zxidLease plus key creation; fencing via revision
ConfigurationZnode data with watchesKeys with watches from a revision

Both expose ordering numbers (zxid, revision) that work as fencing tokens (Day 18). In the ZooKeeper lock recipe, each waiter watches only the znode just before its own rather than the lock as a whole, so a release wakes one client, not all of them at once (the herd effect).

An etcd lock, step by step

Here's the lock recipe with etcdctl, tested against a single etcd 3.6 node. First, a lease that expires after 10 seconds unless the holder renews it:

SQL
etcdctl lease grant 10
# lease 694da166a51d5005 granted with TTL(10s)

Then a transaction: if the lock key has never been created, create it attached to the lease; otherwise, read who holds it. The input is the condition, a blank line, the operations to run on success, a blank line, and the operations to run on failure:

Code
etcdctl txn <<'EOF'
create("/locks/report") = "0"

put /locks/report worker-a --lease=694da166a51d5005

get /locks/report

EOF
Code
worker-a:  SUCCESS        (the key is created; worker-a holds the lock)
worker-b:  FAILURE        (the same transaction runs the failure branch instead)
           /locks/report
           worker-a

create("/locks/report") = "0" means "this key doesn't exist", because a missing key's creation revision is 0. The check and the write happen atomically, so two workers can't both succeed. The key's create_revision (2 here, from etcdctl get /locks/report -w json) is the fencing token: a later holder's key is always created at a higher revision. The holder keeps the lease alive with etcdctl lease keep-alive; if it crashes, the key disappears 10 seconds later and the next worker's transaction succeeds. etcdctl lock and the client libraries' concurrency packages wrap this recipe, and queue waiters in revision order, the same way ZooKeeper uses sequential znodes.

Operating ZooKeeper and etcd

  • 3 or 5 nodes, on separate machines and zones. More nodes don't mean more throughput; every write goes through the leader to a majority.
  • Fast disks matter: each write is fsynced to the log before acknowledging. Slow disks cause leader elections and timeouts in Kubernetes.
  • Keep data small: they're for coordination metadata (kilobytes per key), not application data. etcd defaults to a 2 GB storage quota and periodic compaction of old revisions.
  • Back up regularly (etcd snapshot save), and practice restoring; losing the Kubernetes etcd means losing the cluster's state.
  • Watch for large numbers of watches or clients reconnecting at once after an incident.

Check yourself

They're deleted automatically when the client's session ends, for example when it crashes or loses its connection for too long. A crashed leader's or service's entry disappears on its own, and watchers are notified to elect a new leader or update their routing.

A global, increasing version for every change, used to read history, resume watches without missing events, and act as a fencing token.

They're designed for small, critical coordination data. Every write goes through consensus and is stored on every node, and etcd keeps the whole keyspace in memory, so large or frequently changing data hurts performance and stability for everyone using the cluster.

It checks that the key doesn't exist yet, since a missing key's creation revision is 0; the key's create_revision, which is higher for every later holder, is the fencing token.

ZooKeeper Atomic Broadcast, the consensus protocol ZooKeeper uses to elect a leader and replicate writes in order to a majority.