Catalog My learning Blog
Raft and Paxos Explained: Consensus, ZooKeeper and etcd · Phase 3 · Lesson 12 · 10 min read

What Is a Coordination Service?

On this page
  1. Learning Objective
  2. Concept Explanation
  3. Worked Example: The Same Leader Election in Both Services
  4. Real-World Examples
  5. Review Questions & Answers

Learning Objective

Describe the primitives a coordination service layers on top of raw consensus - leader election, distributed locks, configuration storage, membership, and service discovery - and explain why these services are deliberately built as small, low-throughput, strongly consistent stores rather than as general databases.

Concept Explanation

Raft and Paxos give you an agreed-upon ordered log and nothing else. That is a powerful primitive and a poor developer interface: nobody wants to think about commit indices and AppendEntries when what they need is "make sure exactly one of these twelve workers runs the nightly job". A coordination service closes that gap. ZooKeeper, etcd, and Consul each run a consensus algorithm internally - ZAB, Raft, and Raft respectively - and expose it as a small, strongly consistent data store with a handful of extra features that turn out to be exactly what the classic coordination problems need.

Those extra features are few and carefully chosen. The first is a compare-and-set primitive, so a client can write a key only if its current version matches what the client last read, which makes atomic state transitions possible without holding any lock. The second is ephemeral or lease-bound keys, which the service deletes automatically when the owning client's session expires, and this is the feature that makes liveness detection possible without any application-level heartbeat logic. The third is watches or a streaming change feed, so a client learns about a change within milliseconds instead of polling. The fourth is ordering, whether through sequential node naming in ZooKeeper or a global revision counter in etcd, which gives every change a position in a total order. Given those four, every coordination recipe becomes short. Leader election is an ephemeral key created with compare-and-set, where the winner is whoever creates it and everyone else watches it. Membership is a set of ephemeral keys under a common prefix, and a node crashing removes its own key automatically. Configuration distribution is a key plus a watch.

The deliberate design decision that surprises people is how small these systems are meant to be. ZooKeeper keeps its entire dataset in memory on every server and enforces a default one-megabyte limit per znode. etcd ships with a default backend size quota of two gigabytes and warns against exceeding it. Both are limits that would be absurd in a database and are entirely reasonable here, because every write must be agreed by a majority and applied by every server, so write throughput is bounded by consensus rather than by storage, typically landing in the low tens of thousands of writes per second regardless of hardware. A coordination service is engineered for a modest volume of extremely important data, not a large volume of ordinary data.

The most common architectural mistake follows directly from ignoring that. Teams discover a service that is consistent, highly available, and offers watches, and reach for it as a job queue, a session store, a metrics sink, or a general-purpose configuration database holding per-request state. Each of these produces high write churn against a system where every write is a consensus round replicated to every node and retained in history until compaction. The characteristic failure is not a gradual slowdown but a cliff: etcd hits its backend quota, raises a no-space alarm, and switches the entire cluster to read-only, which in a Kubernetes cluster means the control plane stops accepting any change at all until an operator compacts and defragments. The rule to internalise is that data belongs in a coordination service only if the whole cluster must agree on it and it changes infrequently.

It is also worth being clear about what a coordination service does not give you for free, because the failure here is subtle. Holding a lock from a coordination service does not prevent a client from acting after the lock has expired, because the client may be paused in garbage collection or partitioned when the expiry happens and will not learn about it until it resumes. The service knows the lock is gone; the client does not. Making a lock genuinely safe requires the protected resource to participate by rejecting stale holders, which is the fencing-token pattern covered in its own lesson later in this phase. Likewise, a watch tells you something changed, not what the current value is, so the correct pattern is always to re-read after a watch fires rather than treating the notification as a data delivery mechanism.

Coordination NeedPrimitive UsedZooKeeper Formetcd Form
Leader electionEphemeral key plus orderingEphemeral sequential znode, lowest sequence winsKey with a lease, claimed by transaction on create revision
Membership and livenessEphemeral keys under a prefixEphemeral znodes under a parent, children listedKeys attached to a lease, ranged by prefix
Configuration distributionKey plus change notificationZnode data plus a one-shot watchKey plus a streaming watch from a revision
Distributed lockExclusive creation plus orderingEphemeral sequential znode with predecessor watchingLease-bound key acquired by transaction
Atomic state transitionCompare-and-setsetData with an expected versionTxn comparing mod_revision or value

Worked Example: The Same Leader Election in Both Services

Seeing one recipe expressed in both systems shows how similar the underlying idea is and how differently the two APIs express it, which matters when porting between them or reading someone else's coordination code.

Python
# --- ZooKeeper: ephemeral sequential znodes ---
# Every candidate creates one node; the lowest sequence number wins.

path = zk.create('/election/candidate-',
                 data=my_id,
                 ephemeral=True, sequence=True)
# -> '/election/candidate-0000000037'

def check_leadership():
    children = sorted(zk.get_children('/election'))
    me = path.split('/')[-1]
    if children[0] == me:
        become_leader()
        return
    # NOT the leader. Watch ONLY the node immediately before me, never
    # the whole parent: if all 500 candidates watched /election, one
    # departure would wake all 500 at once (the herd effect).
    predecessor = children[children.index(me) - 1]
    zk.exists('/election/' + predecessor, watch=on_predecessor_gone)

def on_predecessor_gone(event):
    check_leadership()      # ZooKeeper watches are ONE-SHOT: re-arm

# If this process dies or its session expires, the ephemeral node is
# deleted by the server, the successor's watch fires, and it takes over.
# No heartbeat logic in the application at all.


# --- etcd: a lease plus a conditional transaction ---
lease = etcd.lease_grant(ttl=10)          # seconds
etcd.lease_keep_alive(lease)              # background refresh

# "Create /election/leader ONLY if it does not already exist."
# create_revision == 0 is etcd's idiom for "key absent".
resp = etcd.txn(
    compare=[Compare(key='/election/leader', target='create_revision', value=0)],
    success=[Put('/election/leader', my_id, lease=lease)],
    failure=[Get('/election/leader')],
)

if resp.succeeded:
    become_leader()
else:
    current_leader = resp.responses[0].value
    # Watches in etcd are STREAMING from a revision, not one-shot,
    # so no re-arming is needed and no events are missed in between.
    for event in etcd.watch('/election/leader', start_revision=resp.header.revision):
        if event.type == 'DELETE':
            retry_election()

# If this process dies, lease_keep_alive stops, the lease expires after
# at most 10s, etcd deletes the key, and every watcher is notified.

# THE SHARED CAVEAT: winning either election does not make it safe to
# act as leader indefinitely. A GC pause longer than the session or
# lease timeout means the service has already revoked leadership while
# this process still believes it holds it. Safety requires the protected
# resource to reject stale holders - see the fencing-token lesson.
A coordination service as a thin layer over consensus: a replicated log underneath, and above it the four primitives that make coordination recipes short, with leader election, membership, configuration, and locking each shown as a small composition of those primitives.
A coordination service as a thin layer over consensus: a replicated log underneath, and above it the four primitives that make coordination recipes short, with leader election, membership, configuration, and locking each shown as a small composition of those primitives.

Real-World Examples

  • Apache Hadoop uses ZooKeeper for HDFS NameNode high availability through the ZKFailoverController, which holds an ephemeral znode representing active status: if the active NameNode process dies or its session expires, the znode disappears and the standby is notified and promotes itself. This is the ephemeral-key leader election recipe applied to one of the most consequential single points of failure in a Hadoop cluster, and it is the reason a production HDFS HA deployment requires a ZooKeeper ensemble alongside it.
  • Apache Kafka used ZooKeeper for broker registration, controller election, and topic metadata from its earliest releases until KRaft mode became the default in Kafka 3.3 and ZooKeeper support was removed entirely in Kafka 4.0. The migration was driven substantially by scale limits: metadata for very large clusters strained ZooKeeper's design point of a small in-memory dataset, and the two-system operational burden was significant, which is a direct illustration of both why coordination services are used and where their intentional smallness becomes a constraint.
Key TakeawaysA coordination service is a small, strongly consistent store with four extra primitives - compare-and-set, ephemeral or lease-bound keys, watches, and ordering - and those four are enough to express every classic coordination recipe. It is not a database, a queue, or a session store: every write is a consensus round applied by every node, so write throughput tops out in the low tens of thousands per second and the total dataset is deliberately capped. Put data there only when the entire cluster must agree on it and it changes infrequently.

Review Questions & Answers

Compare-and-set, which allows atomic state transitions without any lock by making a write conditional on the version last read. Ephemeral or lease-bound keys, which the service removes when the owning session expires and which therefore provide crash detection without any application-level heartbeat code. Watches or a streaming change feed, which let clients learn about changes in milliseconds instead of polling and thereby make configuration distribution and failover detection practical. Ordering, through sequential naming or a global revision counter, which gives every change a position in a total order and is what makes fair queueing and fencing tokens possible. Remove any one and the standard recipes either become impossible or degrade into polling loops with race conditions.

Because a queue is high-churn by definition and a coordination service is engineered for low write volume. Every enqueue and dequeue would be a full consensus round replicated to every node, durably written, applied by every server, and retained in history until compaction. Throughput would be capped in the low tens of thousands of operations per second regardless of hardware, and the accumulated data would drive the store toward its size limit. The failure mode is abrupt rather than gradual: etcd raises a no-space alarm and turns the whole cluster read-only, which in a Kubernetes cluster freezes the entire control plane. Queues belong in systems designed for throughput and retention, with the coordination service used only for the small agreed facts about who is processing what.

To avoid the herd effect. A watch on the parent fires for every child change, so if five hundred candidates all watched the parent, one node departing would wake all five hundred simultaneously, each of which would then list children and re-register a watch, producing a burst of load on the ensemble that grows with the number of candidates. Watching only the immediate predecessor means exactly one client is woken per departure, which is the one that actually needs to act because it may now be the lowest sequence number. The total work per failover becomes constant rather than proportional to the number of candidates.

Because expiry is observed by the service, not by the client. A client holding a lock can be paused by a long garbage collection, descheduled, or partitioned, and during that pause its session times out and the service grants the lock to someone else. When the paused client resumes it has no immediate way to know this and proceeds to write, so two clients act concurrently despite the service having behaved perfectly. Closing the gap requires the protected resource to participate: the lock service issues a monotonically increasing fencing token with each grant, and the resource rejects any write carrying a token lower than the highest it has already accepted.

Because a notification reports that something changed, not what the value is now, and several changes can occur between the event being generated and the client acting on it. In ZooKeeper this is acute because watches are one-shot: the watch is removed when it fires, so any further changes before the client re-registers are not delivered at all, and only a re-read reveals the current state. etcd's streaming watches do deliver every intermediate event from a given revision, but even there the client should treat the stream as a trigger to reconcile against current state rather than as an authoritative incremental feed, since a stream can be compacted out from under a slow consumer and must then be restarted from a fresh read.