Catalog My learning Blog
System Design Roadmap: 100-Day Course for Engineers · Phase 7 · Lesson 63 · 5 min read

Leader Election Patterns in Distributed Systems

On this page
  1. The billing job that ran three times
  2. What is leader election?
  3. How leader election works: terms and heartbeats
  4. The catch: you can't tell slow from dead
  5. Leader election with a lease: Kubernetes, etcd or a database
  6. Check yourself

The billing job that ran three times

A scheduled job charges subscriptions at midnight. The app runs on three instances for availability, and each instance has the scheduler built in. At midnight, all three ran the job, and customers were charged three times. The fix isn't to run one instance (then there's no availability). It's to make the instances elect a leader that alone runs the job, with another taking over if it dies.

What is leader election?

In distributed systems, leader election picks exactly one node among peers to coordinate something: process scheduled jobs, accept writes for a partition, assign work to others. One decision-maker avoids conflicting decisions. The challenge is doing it with no central referee, while nodes crash and networks split.

How leader election works: terms and heartbeats

The common pattern, used by Raft (Day 62):

  1. Time is divided into numbered terms. Each term has at most one leader.
  2. The leader sends regular heartbeats to followers.
  3. A follower that hears no heartbeat within its election timeout assumes the leader is gone, increments the term, votes for itself and asks the others for votes.
  4. A node votes for at most one candidate per term. A candidate with a majority of votes becomes leader for that term.
  5. Any node that sees a higher term steps down and follows.
A leader sends heartbeats to followers; a missed heartbeat triggers a new election with a higher term number.
A leader sends heartbeats to followers; a missed heartbeat triggers a new election with a higher term number.

Randomized timeouts. If every follower waited exactly 300 ms, they'd all become candidates at once, split the vote, and repeat. Raft uses random timeouts, for example 150 to 300 ms, so one node usually starts first and wins.

The catch: you can't tell slow from dead

A missing heartbeat means "slow or dead", and you can't tell which. A leader paused by a long garbage collection, or cut off by a network partition, may still believe it's leader while the others elect a new one. For a moment, two nodes think they're leader.

Defenses:

  • Majorities. The old leader can't commit anything without a majority, which now follows the new leader.
  • Term numbers on every action. Receivers reject commands from an older term.
  • Fencing tokens. The leader's term or lease number is passed to storage systems, which reject writes carrying an older number (Day 64).
  • Leader step-down. A leader that can't reach a majority within its lease period stops acting as leader.

Leader election with a lease: Kubernetes, etcd or a database

You rarely implement elections yourself. Use a coordination service and a lease: a lock with an expiry that the holder keeps renewing.

  • Kubernetes: a Lease object in the API server. Client libraries (client-go's leader election, and equivalents in other languages) handle acquire, renew and step-down. Kubernetes' own controllers elect leaders this way.
  • etcd or ZooKeeper: election recipes built on their consensus.
  • A database row: one statement both takes an expired lease and renews your own, and the database's clock is the only clock involved:
SQL
UPDATE leader SET holder = 'node-b', expires_at = now() + interval '15 seconds'
WHERE name = 'billing' AND (expires_at < now() OR holder = 'node-b');
-- 1 row updated: node-b is leader for the next 15 seconds.  0 rows: someone else is.

It's simple, and adequate for many job schedulers.

  • Job frameworks: Hangfire, Quartz clustering and similar tools use database locks so a scheduled job runs once across instances.

For the billing job: each instance tries to acquire the billing-leader lease every few seconds. The holder runs the job and renews the lease; if it dies, the lease expires and another instance takes over. And the billing job itself is idempotent (Day 65), so a rare double run during a takeover charges no one twice.

Get the timing right. The leader must stop acting before its lease could expire, not after, so it renews well inside the lease and steps down if renewal keeps failing. Kubernetes' own controllers use a 15-second lease that must be renewed within 10 seconds, retried every 2. And the leader measures "time since I last renewed" with a monotonic clock (Day 61); a wall-clock jump could otherwise make it think it still holds a lease that has run out.

Check yourself

If every follower used the same timeout, they'd all notice a missing leader at the same moment, all start elections, and split the vote so nobody wins, over and over. Random timeouts mean one follower usually times out first, asks for votes before the others start, and wins the election.

Renewing so close to expiry leaves no margin: a slow renewal or a short pause lets the lease lapse while the leader keeps working. Renew well inside the lease and stop work when renewal fails.

The old leader may be paused or partitioned and not yet know a new leader was elected. Majorities, term checks and fencing tokens keep its actions from taking effect.

Have instances compete for a lease in Kubernetes, etcd or the database; only the holder runs the job, and make the job idempotent as a backstop.

During the pause its lease may have expired and a new leader been elected, but the old leader doesn't know that time passed. When it resumes, it may act as leader again, so two leaders do the same work. Fencing tokens, checked by the resource being written to, reject the stale leader's actions.