What Is a Coordination Service?
On this page
Learning Objective
Describe the primitives a coordination service layers on top of raw consensus - leader election, distributed locks, configuration storage, membership, and service discovery - and explain why these services are deliberately built as small, low-throughput, strongly consistent stores rather than as general databases.
Concept Explanation
Raft and Paxos give you an agreed-upon ordered log and nothing else. That is a powerful primitive and a poor developer interface: nobody wants to think about commit indices and AppendEntries when what they need is "make sure exactly one of these twelve workers runs the nightly job". A coordination service closes that gap. ZooKeeper, etcd, and Consul each run a consensus algorithm internally - ZAB, Raft, and Raft respectively - and expose it as a small, strongly consistent data store with a handful of extra features that turn out to be exactly what the classic coordination problems need.
Those extra features are few and carefully chosen. The first is a compare-and-set primitive, so a client can write a key only if its current version matches what the client last read, which makes atomic state transitions possible without holding any lock. The second is ephemeral or lease-bound keys, which the service deletes automatically when the owning client's session expires, and this is the feature that makes liveness detection possible without any application-level heartbeat logic. The third is watches or a streaming change feed, so a client learns about a change within milliseconds instead of polling. The fourth is ordering, whether through sequential node naming in ZooKeeper or a global revision counter in etcd, which gives every change a position in a total order. Given those four, every coordination recipe becomes short. Leader election is an ephemeral key created with compare-and-set, where the winner is whoever creates it and everyone else watches it. Membership is a set of ephemeral keys under a common prefix, and a node crashing removes its own key automatically. Configuration distribution is a key plus a watch.
The deliberate design decision that surprises people is how small these systems are meant to be. ZooKeeper keeps its entire dataset in memory on every server and enforces a default one-megabyte limit per znode. etcd ships with a default backend size quota of two gigabytes and warns against exceeding it. Both are limits that would be absurd in a database and are entirely reasonable here, because every write must be agreed by a majority and applied by every server, so write throughput is bounded by consensus rather than by storage, typically landing in the low tens of thousands of writes per second regardless of hardware. A coordination service is engineered for a modest volume of extremely important data, not a large volume of ordinary data.
The most common architectural mistake follows directly from ignoring that. Teams discover a service that is consistent, highly available, and offers watches, and reach for it as a job queue, a session store, a metrics sink, or a general-purpose configuration database holding per-request state. Each of these produces high write churn against a system where every write is a consensus round replicated to every node and retained in history until compaction. The characteristic failure is not a gradual slowdown but a cliff: etcd hits its backend quota, raises a no-space alarm, and switches the entire cluster to read-only, which in a Kubernetes cluster means the control plane stops accepting any change at all until an operator compacts and defragments. The rule to internalise is that data belongs in a coordination service only if the whole cluster must agree on it and it changes infrequently.
It is also worth being clear about what a coordination service does not give you for free, because the failure here is subtle. Holding a lock from a coordination service does not prevent a client from acting after the lock has expired, because the client may be paused in garbage collection or partitioned when the expiry happens and will not learn about it until it resumes. The service knows the lock is gone; the client does not. Making a lock genuinely safe requires the protected resource to participate by rejecting stale holders, which is the fencing-token pattern covered in its own lesson later in this phase. Likewise, a watch tells you something changed, not what the current value is, so the correct pattern is always to re-read after a watch fires rather than treating the notification as a data delivery mechanism.
| Coordination Need | Primitive Used | ZooKeeper Form | etcd Form |
|---|---|---|---|
| Leader election | Ephemeral key plus ordering | Ephemeral sequential znode, lowest sequence wins | Key with a lease, claimed by transaction on create revision |
| Membership and liveness | Ephemeral keys under a prefix | Ephemeral znodes under a parent, children listed | Keys attached to a lease, ranged by prefix |
| Configuration distribution | Key plus change notification | Znode data plus a one-shot watch | Key plus a streaming watch from a revision |
| Distributed lock | Exclusive creation plus ordering | Ephemeral sequential znode with predecessor watching | Lease-bound key acquired by transaction |
| Atomic state transition | Compare-and-set | setData with an expected version | Txn comparing mod_revision or value |
Worked Example: The Same Leader Election in Both Services
Seeing one recipe expressed in both systems shows how similar the underlying idea is and how differently the two APIs express it, which matters when porting between them or reading someone else's coordination code.
# --- ZooKeeper: ephemeral sequential znodes ---
# Every candidate creates one node; the lowest sequence number wins.
path = zk.create('/election/candidate-',
data=my_id,
ephemeral=True, sequence=True)
# -> '/election/candidate-0000000037'
def check_leadership():
children = sorted(zk.get_children('/election'))
me = path.split('/')[-1]
if children[0] == me:
become_leader()
return
# NOT the leader. Watch ONLY the node immediately before me, never
# the whole parent: if all 500 candidates watched /election, one
# departure would wake all 500 at once (the herd effect).
predecessor = children[children.index(me) - 1]
zk.exists('/election/' + predecessor, watch=on_predecessor_gone)
def on_predecessor_gone(event):
check_leadership() # ZooKeeper watches are ONE-SHOT: re-arm
# If this process dies or its session expires, the ephemeral node is
# deleted by the server, the successor's watch fires, and it takes over.
# No heartbeat logic in the application at all.
# --- etcd: a lease plus a conditional transaction ---
lease = etcd.lease_grant(ttl=10) # seconds
etcd.lease_keep_alive(lease) # background refresh
# "Create /election/leader ONLY if it does not already exist."
# create_revision == 0 is etcd's idiom for "key absent".
resp = etcd.txn(
compare=[Compare(key='/election/leader', target='create_revision', value=0)],
success=[Put('/election/leader', my_id, lease=lease)],
failure=[Get('/election/leader')],
)
if resp.succeeded:
become_leader()
else:
current_leader = resp.responses[0].value
# Watches in etcd are STREAMING from a revision, not one-shot,
# so no re-arming is needed and no events are missed in between.
for event in etcd.watch('/election/leader', start_revision=resp.header.revision):
if event.type == 'DELETE':
retry_election()
# If this process dies, lease_keep_alive stops, the lease expires after
# at most 10s, etcd deletes the key, and every watcher is notified.
# THE SHARED CAVEAT: winning either election does not make it safe to
# act as leader indefinitely. A GC pause longer than the session or
# lease timeout means the service has already revoked leadership while
# this process still believes it holds it. Safety requires the protected
# resource to reject stale holders - see the fencing-token lesson.Real-World Examples
- Apache Hadoop uses ZooKeeper for HDFS NameNode high availability through the ZKFailoverController, which holds an ephemeral znode representing active status: if the active NameNode process dies or its session expires, the znode disappears and the standby is notified and promotes itself. This is the ephemeral-key leader election recipe applied to one of the most consequential single points of failure in a Hadoop cluster, and it is the reason a production HDFS HA deployment requires a ZooKeeper ensemble alongside it.
- Apache Kafka used ZooKeeper for broker registration, controller election, and topic metadata from its earliest releases until KRaft mode became the default in Kafka 3.3 and ZooKeeper support was removed entirely in Kafka 4.0. The migration was driven substantially by scale limits: metadata for very large clusters strained ZooKeeper's design point of a small in-memory dataset, and the two-system operational burden was significant, which is a direct illustration of both why coordination services are used and where their intentional smallness becomes a constraint.
Review Questions & Answers
Compare-and-set, which allows atomic state transitions without any lock by making a write conditional on the version last read. Ephemeral or lease-bound keys, which the service removes when the owning session expires and which therefore provide crash detection without any application-level heartbeat code. Watches or a streaming change feed, which let clients learn about changes in milliseconds instead of polling and thereby make configuration distribution and failover detection practical. Ordering, through sequential naming or a global revision counter, which gives every change a position in a total order and is what makes fair queueing and fencing tokens possible. Remove any one and the standard recipes either become impossible or degrade into polling loops with race conditions.
Because a queue is high-churn by definition and a coordination service is engineered for low write volume. Every enqueue and dequeue would be a full consensus round replicated to every node, durably written, applied by every server, and retained in history until compaction. Throughput would be capped in the low tens of thousands of operations per second regardless of hardware, and the accumulated data would drive the store toward its size limit. The failure mode is abrupt rather than gradual: etcd raises a no-space alarm and turns the whole cluster read-only, which in a Kubernetes cluster freezes the entire control plane. Queues belong in systems designed for throughput and retention, with the coordination service used only for the small agreed facts about who is processing what.
To avoid the herd effect. A watch on the parent fires for every child change, so if five hundred candidates all watched the parent, one node departing would wake all five hundred simultaneously, each of which would then list children and re-register a watch, producing a burst of load on the ensemble that grows with the number of candidates. Watching only the immediate predecessor means exactly one client is woken per departure, which is the one that actually needs to act because it may now be the lowest sequence number. The total work per failover becomes constant rather than proportional to the number of candidates.
Because expiry is observed by the service, not by the client. A client holding a lock can be paused by a long garbage collection, descheduled, or partitioned, and during that pause its session times out and the service grants the lock to someone else. When the paused client resumes it has no immediate way to know this and proceeds to write, so two clients act concurrently despite the service having behaved perfectly. Closing the gap requires the protected resource to participate: the lock service issues a monotonically increasing fencing token with each grant, and the resource rejects any write carrying a token lower than the highest it has already accepted.
Because a notification reports that something changed, not what the value is now, and several changes can occur between the event being generated and the client acting on it. In ZooKeeper this is acute because watches are one-shot: the watch is removed when it fires, so any further changes before the client re-registers are not delivered at all, and only a re-read reveals the current state. etcd's streaming watches do deliver every intermediate event from a given revision, but even there the client should treat the stream as a trigger to reconcile against current state rather than as an authoritative incremental feed, since a stream can be compacted out from under a slow consumer and must then be restarted from a fresh read.