Distributed Locks with ZooKeeper and etcd
On this page
Why a lock statement stops at the process boundary
Within one process, a lock statement or mutex keeps two threads out of the same critical section. Run the app on three machines and each has its own lock object; nothing stops machine A and machine B from both entering. To coordinate across machines, you need something all of them can see.
A distributed lock is held in a shared coordination system. Day 10 of the Redis course and Day 63 here touched on it; this lesson covers the properties that make it safe or unsafe.
Distributed locks are leases
If a lock holder crashes, the lock must not stay held forever. So distributed locks are leases: they expire unless renewed. The holder renews periodically (a heartbeat); if it dies, the lease lapses and someone else can acquire it.
How ZooKeeper and etcd implement locks
ZooKeeper uses ephemeral sequential nodes. Each client wanting the lock creates a node like /locks/billing/lock-0000000042. Ephemeral means it disappears automatically when the client's session ends. Sequential means ZooKeeper appends an increasing number. The client with the lowest number holds the lock; each other client watches the node just before its own, so when that one disappears, the next in line is notified. No thundering herd, and a fair queue.
etcd uses leases attached to keys: create a lease with a TTL, write the lock key with that lease (only if the key doesn't exist), and keep the lease alive. If the holder stops renewing, the lease expires and the key disappears. etcd's client libraries provide a ready lock API.
Both run consensus underneath (Day 62), so the lock state survives node failures and doesn't split during a partition.
The danger: a lock you no longer hold
Here's the sequence every lock design must handle:
- Client A acquires the lock with a 10-second lease.
- A pauses for 15 seconds: a long garbage collection, a VM migration, a slow disk.
- The lease expires. Client B acquires the lock and starts writing.
- A wakes up, still believing it holds the lock, and writes too.
Renewal doesn't help, because A was paused and couldn't renew or notice. Only the resource being protected can stop A.
Fencing tokens: stopping a stale lock holder
Every time the lock is granted, the coordination service issues an increasing number: a fencing token. ZooKeeper's sequence number or etcd's revision works. The client passes the token with every write. The storage system remembers the highest token it has seen and rejects writes with a lower one.
A acquires the lock, token 33 -> pauses
B acquires the lock, token 34 -> writes with 34 (storage records 34)
A wakes up -> writes with 33 (rejected: 33 < 34)This requires the protected resource to check tokens, for example a WHERE fence_token < :token condition on updates, or a version column. Without it, any lease-based lock can be violated by a long enough pause.
Choosing a lock: Redis, a database or etcd
Redis locks deserve a specific note. A Redis SET key value NX PX 10000 lock has an expiry but no fencing token, so it's exposed to exactly the paused-client problem above. Martin Kleppmann's 2016 analysis of the multi-node Redlock algorithm argues it shouldn't be used where correctness depends on the lock, and the debate that followed is worth reading (Kleppmann, 2016). Use Redis locks for efficiency, not safety.
Your database may already be the simplest lock service. PostgreSQL has advisory locks (pg_try_advisory_lock), SQL Server has sp_getapplock, and the .NET DistributedLock library wraps these and others behind one API. Note that session-level advisory locks don't work through PgBouncer in transaction mode (Day 28).
| Need | Use |
|---|---|
| Avoid duplicate work, where a rare duplicate is harmless | A simple Redis lock with expiry, or a database lock |
| Correctness, where two holders would corrupt data | etcd or ZooKeeper, plus fencing tokens checked by storage |
| Run a scheduled job once across instances | Kubernetes lease, job framework's database lock, plus an idempotent job |
Often the best lock is none: a database constraint, an idempotency key (Day 65) or partitioning work so each item has one owner can remove the need entirely.
Check yourself
Each process has its own lock object in its own memory, so a lock taken in instance A means nothing to instances B and C. Nothing coordinates between machines. Mutual exclusion across instances needs a shared coordinator, such as a database row lock, a Redis lease or etcd.
A paused holder can resume after its lock passed to another client, and Redis locks have no fencing token. Use a database transaction or version check on the balance itself, or a fenced lock the storage verifies.
It paused, for example during garbage collection, long enough for its lease to expire and another client to acquire the lock, then resumed believing it still held it.
Each lock grant gets a higher token, writes carry it, and storage rejects writes with a token lower than the highest it has seen.
When the protected work already writes to that database. A row lock with SELECT ... FOR UPDATE, or an advisory lock, ties the lock to the same transaction as the data change, so there's no separate system to fail and no lease that can expire mid-write.