Catalog My learning Blog
Raft and Paxos Explained: Consensus, ZooKeeper and etcd · Phase 3 · Lesson 13 · 11 min read

ZooKeeper Explained: Znodes, Watches, Sessions

On this page
  1. Learning Objective
  2. Concept Explanation
  3. Worked Example: A Correct Watch Loop and the Race a Naive One Creates
  4. Real-World Examples
  5. Review Questions & Answers

Learning Objective

Navigate ZooKeeper's znode model including ephemeral and sequential types, explain the one-shot watch semantics and the races they create, and describe how sessions, tick time, and the ZAB protocol determine when an ephemeral node actually disappears.

Concept Explanation

ZooKeeper presents a hierarchical namespace that looks like a filesystem: paths such as /services/payments/leader address znodes, each of which holds a small data payload and may have children. The filesystem resemblance is deliberate and also misleading, because every znode can hold data and have children simultaneously, there are no partial reads or writes, and the whole dataset is held in memory on every server. The default per-znode data limit is one megabyte, and the practical guidance is to stay far below it: znodes are for identifiers, addresses, and small configuration blobs, not for content.

Znode type is the feature that makes the coordination recipes work. A persistent znode remains until explicitly deleted. An ephemeral znode is bound to the session of the client that created it and is deleted automatically by the server when that session ends, whether through a clean close, a crash, or a timeout, and ephemeral znodes may not have children. A sequential znode has a monotonically increasing ten-digit counter appended to its name by the server, so a client requesting /queue/item- receives /queue/item-0000000004, with the counter maintained by the parent and therefore totally ordered across all clients. Combining the two gives the ephemeral sequential znode, which is the single most useful object in ZooKeeper: it provides an ordered position that vanishes automatically when its owner dies, and it is the foundation of both leader election and distributed locking.

Watches are the notification mechanism and their semantics cause more bugs than anything else in the API. A watch is set as a side effect of a read operation - exists, getData, or getChildren - and it fires exactly once, after which it is removed. The event tells you the type of change and the path, and deliberately does not include the new data. Between the watch firing and the client re-registering, further changes occur unobserved, so a client that treats watch events as a complete change stream will miss updates. The correct pattern is always the same: on any event, re-read the current state and set a new watch in the same operation, and write the handler so that processing the same state twice is harmless. ZooKeeper 3.6 added persistent recursive watches through addWatch, which removes the re-arming burden, but a great deal of existing code and nearly all published recipes assume the one-shot model.

Sessions are where ZooKeeper's failure detection actually lives, and their timing is more subtle than it first appears. A client establishes a session with a negotiated timeout, bounded by the server between two and twenty times the configured tickTime, which defaults to two seconds and therefore yields a default range of four to forty seconds. The client library sends heartbeats and, importantly, can transparently migrate its session to a different server in the ensemble if the one it was connected to fails, so a server crash does not by itself expire a session. Expiry is declared by the ensemble, not by the client, and only the ensemble can delete the ephemeral nodes that expiry implies. The consequence that trips people up is the disconnect period: while a client is disconnected but its session has not yet expired, it holds a stale view and must not assume it still owns its ephemeral nodes, because it will not find out whether the session survived until it reconnects or the timeout elapses. Correct clients treat the Disconnected state as an immediate signal to stop acting as leader, not as a transient blip.

Underneath, ZooKeeper runs ZAB, the ZooKeeper Atomic Broadcast protocol, which is a leader-based total-order broadcast closely related to Raft in structure and to Multi-Paxos in lineage. Every transaction carries a zxid, a sixty-four-bit identifier whose high thirty-two bits are the leader epoch and whose low thirty-two bits are a counter within that epoch, giving every change a globally ordered and comparable position. All writes are routed to the leader and require acknowledgement from a quorum; reads are served directly from whichever server the client is connected to, which makes reads fast and locally scalable but also means a read can return data older than a write that already completed elsewhere. ZooKeeper does guarantee that a single client sees its own operations in order and never goes backwards in time, and when a client needs a genuinely current read it calls sync, which forces the connected server to catch up with the leader before the subsequent read is served.

Znode TypeLifetimeMay Have ChildrenPrimary Use
PersistentUntil explicitly deletedYesConfiguration, namespace parents, durable metadata
EphemeralUntil the owning session endsNoLiveness and membership: presence means the owner is alive
Persistent sequentialUntil explicitly deleted, name carries an ordered counterYesDurable ordered records such as a barrier or queue position
Ephemeral sequentialUntil the owning session ends, with an ordered counterNoLeader election and distributed locks: the canonical recipe

Worked Example: A Correct Watch Loop and the Race a Naive One Creates

Almost every ZooKeeper bug in production traces back to treating a one-shot watch as a durable subscription, so it is worth seeing the broken version, the race it creates, and the shape of the fix.

Python
# --- BROKEN: treating watch events as a change stream ---
def on_config_change(event):
    new_config = event.data          # NO. Events carry no data at all.
    apply(new_config)
    # No new watch registered -> this handler never fires again.

zk.get_data('/config/app', watch=on_config_change)

# Two independent bugs:
#   1. ZooKeeper watch events carry the path and event type ONLY.
#      There is no payload to read.
#   2. The watch is consumed by firing. Every subsequent change to
#      /config/app is silently missed, forever.


# --- THE RACE, even in code that does re-register ---
t=0    client calls get_data('/config/app', watch=cb)  -> value "v1"
t=1    config changes to "v2" -> watch FIRES, watch is now REMOVED
t=2    callback begins running
t=3    config changes to "v3"   # NO watch is registered right now
t=4    callback re-reads and gets "v3", re-registers a watch
# The client never observed "v2" as a distinct state. For configuration
# this is fine, because only the latest value matters. For anything
# where each intermediate value must be processed, ZooKeeper watches
# are the wrong tool and the design needs rethinking.


# --- CORRECT: re-read and re-arm atomically, idempotent handling ---
def watch_config():
    # get_data returns the CURRENT value and registers the next watch
    # in the same call, so there is no gap between reading and arming.
    data, stat = zk.get_data('/config/app', watch=lambda e: watch_config())
    apply_config(data, version=stat.version)   # must be idempotent

watch_config()   # initial read arms the first watch


# --- SESSION STATE: the part that actually matters for correctness ---
def on_state_change(state):
    if state == 'CONNECTED':
        watch_config()          # re-arm ALL watches: they are lost on
                                # reconnect and must be re-established
    elif state == 'SUSPENDED':  # disconnected, session NOT yet expired
        stop_acting_as_leader() # IMMEDIATELY. We cannot know whether
                                # our ephemeral node still exists.
    elif state == 'LOST':       # session expired; ephemerals are gone
        rejoin_election()

# The SUSPENDED case is the one teams get wrong. A client that keeps
# acting as leader while disconnected is exactly the scenario where the
# ensemble has already deleted its ephemeral node, another client has
# already won the election, and two processes now believe they lead.


# --- A CURRENT read, when staleness is unacceptable ---
zk.sync('/config/app')    # force this server to catch up to the leader
data, stat = zk.get_data('/config/app')
# Without sync, a read is served from local server state and may be
# older than a write that has already been acknowledged elsewhere.
The ZooKeeper znode tree with persistent, ephemeral, and ephemeral sequential nodes, the one-shot watch cycle from registration through firing to re-arming with its unobserved gap, and the session lifecycle that determines when the ensemble deletes an ephemeral node.
The ZooKeeper znode tree with persistent, ephemeral, and ephemeral sequential nodes, the one-shot watch cycle from registration through firing to re-arming with its unobserved gap, and the session lifecycle that determines when the ensemble deletes an ephemeral node.

Real-World Examples

  • Apache HBase uses ZooKeeper for region server membership and master election, with each region server maintaining an ephemeral znode under /hbase/rs: when a region server process dies or its session expires, the znode disappears, the master is notified through a watch, and the regions that server was hosting are reassigned. This is why HBase documentation places such emphasis on tuning the ZooKeeper session timeout against expected garbage collection pause lengths, since a long stop-the-world pause on a region server expires its session and triggers a needless and expensive region reassignment.
  • Apache Solr in SolrCloud mode stores its entire cluster state, collection configuration, and shard leadership in ZooKeeper, using ephemeral nodes for live node tracking and watches to propagate state changes to every node. Solr's operational guidance recommends a dedicated ZooKeeper ensemble rather than a shared one precisely because coordination latency directly determines how quickly the cluster notices a failed node and reassigns leadership for the affected shards.
Common PitfallZooKeeper watches are one-shot and carry no data: they tell you something changed at a path, they are consumed by firing, and changes occurring before you re-register are never delivered. Always re-read the current value and re-arm the watch in the same call, and make handlers idempotent. Equally important, treat a disconnected-but-not-expired session as an immediate signal to stop acting as leader, because the ensemble may already have deleted your ephemeral node and elected someone else.

Review Questions & Answers

It combines the two properties those recipes need. The sequential part gives every participant a position in a total order maintained by the server, so there is an unambiguous, race-free answer to who is first without any client-side coordination. The ephemeral part ties that position to the creator's session, so a crashed or partitioned participant has its position removed automatically by the ensemble with no heartbeat logic or timeout handling in the application. Together they mean the lowest remaining sequence number is always both well defined and held by a live client, which is exactly the invariant a leader election or a lock queue requires.

It receives exactly one event, generated by the first change, because the watch is consumed by firing and is not re-registered until the client acts. The re-read then returns the third and current value, so the second value is never observed as a distinct state at all. This is acceptable for configuration, where only the latest value matters, and is unacceptable for any design that must process each intermediate state, which is a sign that ZooKeeper watches are the wrong mechanism for that requirement. It is also why handlers must be written to be idempotent and to reconcile against current state rather than to apply deltas.

Because the client cannot observe its own expiry. Session expiry is declared by the ensemble, which then deletes the client's ephemeral node and lets another client win the election, all while the disconnected client sees nothing. A client that continues acting as leader through a disconnection is therefore in exactly the window where a second leader may already be operating. Stopping immediately on suspension is conservative - the session may well survive and leadership resume - but it is the only behaviour that prevents two active leaders, because the alternative depends on information the client provably does not have.

Writes are routed to the leader and require a quorum, but reads are served directly by whichever server the client is connected to, without consulting the leader. That server may not yet have applied the most recent committed transactions, so a read can return a value older than a write already acknowledged elsewhere. ZooKeeper still guarantees a client sees its own operations in order and never moves backwards in time. The sync call forces the connected server to catch up with the leader before the following read is served, converting a potentially stale local read into a current one at the cost of a round trip to the leader.

The paused process sends no heartbeats, so after 40 seconds the ensemble declares the session expired and deletes the region server's ephemeral znode. The master's watch fires, the master concludes the server is dead, and it reassigns every region that server was hosting to other servers, an expensive operation involving log splitting and region reopening. The original process then resumes from its pause, discovers its session is gone, and must shut down or rejoin from scratch. Nothing malfunctioned: the failure detector did exactly what it was configured to do, and a pause longer than the timeout is indistinguishable from a crash. The tuning question is therefore to set the session timeout comfortably above the worst observed pause, accepting slower detection of genuine failures in exchange for not treating routine pauses as deaths.