Catalog My learning Blog
Raft and Paxos Explained: Consensus, ZooKeeper and etcd · Phase 3 · Lesson 14 · 10 min read

etcd and the Kubernetes Control Plane

On this page
  1. Learning Objective
  2. Concept Explanation
  3. Worked Example: Diagnosing a Frozen Control Plane From the etcd Side
  4. Real-World Examples
  5. Review Questions & Answers

Learning Objective

Explain etcd's revision-based MVCC model, leases, and streaming watches, describe precisely how the Kubernetes control plane depends on etcd for every object and every watch, and diagnose the backend quota and compaction failures that most commonly take a control plane down.

Concept Explanation

etcd is a distributed key-value store built on Raft, exposing a gRPC API over a flat, byte-ordered key space. Its most important design choice is multi-version concurrency control keyed by a global revision counter. Every write in the cluster, of any key, increments a single cluster-wide revision, so a revision number is a consistent point-in-time snapshot of the entire store. Each key additionally carries a create revision, a mod revision, and a version count. This is what lets a client read the store as it existed at a specific revision, and, crucially, watch from a specific revision and be guaranteed to receive every change since, which is a materially stronger contract than ZooKeeper's one-shot watches.

Three API primitives do most of the work. Txn is a conditional transaction that compares one or more keys against a create revision, mod revision, version, or value, and executes one block of operations if every comparison holds and another if not, providing compare-and-set without locks. Lease is a time-to-live object that keys can be attached to: the client refreshes it with keep-alive calls, and when refreshes stop, the lease expires and etcd deletes every key attached to it, which is etcd's equivalent of ephemeral znodes with the useful difference that many keys can share one lease. Watch is a long-lived gRPC stream that delivers every event on a key or prefix from a given revision onward, so a client that reconnects can resume exactly where it stopped rather than re-reading everything.

Kubernetes is built on these primitives more literally than most people realise. Every object in the cluster - every pod, service, config map, secret, and custom resource - is a single etcd key under a path like /registry/pods/default/my-app, holding the serialised object. Only the API server talks to etcd; kubelets, schedulers, and controllers never do, which is why etcd credentials are the crown jewels of a cluster and why an unencrypted etcd backup contains every secret in plain form. The resourceVersion that appears on every Kubernetes object is the etcd revision, which is why it is opaque, must not be compared numerically across resource types, and can be used to resume a watch. The controller pattern that defines Kubernetes - list the current state, then watch from that revision for changes - maps directly onto an etcd range read followed by a watch from the returned revision, and that is how thousands of informers stay current without polling.

The most common way a Kubernetes control plane fails is a direct consequence of this design, and it is worth knowing in detail because the symptom is confusing. etcd retains the full history of every revision until an explicit compaction discards old ones, and it ships with a default backend size quota of two gigabytes. A cluster that churns objects rapidly - a CronJob creating thousands of Job and Pod objects, a controller updating status fields in a hot loop, an operator writing frequent lease renewals - accumulates history quickly. When the backend exceeds the quota, etcd raises a NOSPACE alarm and switches the entire cluster to read-only. The visible effect is that every write to the Kubernetes API fails while reads keep working, so the cluster looks partly alive: running pods continue, nothing can be created, scaled, or deleted. Recovery requires compacting old revisions, defragmenting the backend to actually release the disk space, and then explicitly disarming the alarm, because compaction alone reclaims logical space without shrinking the file.

Two further operational realities follow. First, etcd is unusually sensitive to disk latency, because every Raft commit requires an fsync before acknowledgement; the etcd documentation is explicit that the backend commit duration and WAL fsync duration metrics are the primary health signals, and that etcd expects fast local SSDs rather than network-attached storage. A control plane showing intermittent leader elections almost always turns out to have a storage latency problem rather than a network one. Second, because Kubernetes stores everything in etcd, the size and churn of the cluster translate directly into etcd load, which is why the guidance is a three or five member cluster, dedicated disks, and, for large clusters, moving the high-churn events resource to a separate etcd cluster entirely through the API server's events-servers flag.

etcd ConceptWhat It ProvidesKubernetes Use
Global revisionA cluster-wide monotonic counter marking every writeThe resourceVersion on every object; the resume point for every watch
Txn with compareCompare-and-set on revision, version, or valueOptimistic concurrency: a conflicting update fails and the client retries
LeaseA TTL object; attached keys are deleted on expiryNode heartbeats via Lease objects, and leader election for controllers
Streaming watchEvery event from a revision onward, resumableInformers and controllers staying current without polling
CompactionDiscarding history older than a revisionBounding etcd growth; omitting it eventually freezes the control plane

Worked Example: Diagnosing a Frozen Control Plane From the etcd Side

This specific incident recurs often enough to be worth recognising by its symptoms, because the Kubernetes-level error message points nowhere useful and the real diagnosis is two commands deep.

Shell
# SYMPTOM: kubectl reads work fine; every write fails.
$ kubectl get pods                # works
$ kubectl apply -f deploy.yaml
Error from server: etcdserver: mvcc: database space exceeded

# STEP 1: confirm the alarm.
$ etcdctl alarm list
memberID:10276657743932975437 alarm:NOSPACE

$ etcdctl endpoint status --write-out=table
+------------------+---------+---------+-----------+------------+
|     ENDPOINT     |  DB SIZE| IS LEADER| RAFT TERM | RAFT INDEX |
+------------------+---------+---------+-----------+------------+
| https://n1:2379  |  2.1 GB |    true  |        42 | 90412338   |
+------------------+---------+---------+-----------+------------+
# 2.1 GB against a default --quota-backend-bytes of 2 GB.

# WHY: etcd keeps EVERY historical revision until compaction. A CronJob
# running every minute creating Job + Pod objects generates a steady
# stream of revisions that are never discarded if auto-compaction was
# never configured.

# STEP 2: compact history (discard revisions older than current).
$ rev=$(etcdctl endpoint status --write-out=json | jq '.[0].Status.header.revision')
$ etcdctl compact $rev
compacted revision 90412338

# STEP 3: defragment. Compaction frees revisions LOGICALLY; the backend
# file does not shrink until defrag rewrites it. Run per member, one at
# a time - defrag blocks that member for its duration.
$ etcdctl defrag --endpoints=https://n1:2379
$ etcdctl endpoint status --write-out=table   # DB SIZE now ~120 MB

# STEP 4: the alarm does NOT clear itself. Until it is disarmed the
# cluster stays read-only even though space is available.
$ etcdctl alarm disarm
$ kubectl apply -f deploy.yaml    # works again

# PREVENTION, in order of importance:
#   --auto-compaction-retention=1h      # never let history accumulate
#   --quota-backend-bytes=8589934592    # 8 GB for large clusters
#   scheduled defrag during low-traffic windows
#   --events-servers=<separate etcd>    # move the highest-churn resource
#   alert on etcd_mvcc_db_total_size_in_bytes approaching the quota,
#   NOT on the alarm firing - by then the control plane is already frozen

# THE OTHER COMMON etcd INCIDENT: leader election churn.
# Check storage, not the network, first:
#   etcd_disk_wal_fsync_duration_seconds       p99 should be < 10ms
#   etcd_disk_backend_commit_duration_seconds  p99 should be < 25ms
# Every Raft commit needs an fsync before it can be acknowledged, so a
# leader on slow or network-attached storage cannot heartbeat inside the
# election timeout, and leadership moves around the cluster while very
# little commits.
The Kubernetes control plane path into etcd: only the API server connects, every object is a key under /registry, informers list and then watch from a revision, and the revision history grows until compaction and defragmentation reclaim it or the backend quota freezes all writes.
The Kubernetes control plane path into etcd: only the API server connects, every object is a key under /registry, informers list and then watch from a revision, and the revision history grows until compaction and defragmentation reclaim it or the backend quota freezes all writes.

Real-World Examples

  • Kubernetes stores every API object as an etcd key under /registry and exposes the etcd revision directly as each object's resourceVersion, which is why the documented controller pattern is a list followed by a watch from the returned resourceVersion, and why a watch that fails with a resource-too-old error requires a full relist: the requested revision has been compacted away in etcd and the history needed to resume simply no longer exists.
  • The Kubernetes project recommends running events in a separate etcd cluster for large deployments through the API server's events-servers flag, because the Event resource is by far the highest-churn object in a cluster and its write volume can dominate etcd load and history growth. This is an explicit acknowledgement that the coordination service's design point is low write volume, and that one unusually chatty resource can degrade the availability of every other object stored alongside it.
Best PracticeEvery Kubernetes object is an etcd key, and the API server is the only thing that talks to etcd, so etcd health is control plane health. Configure auto-compaction from day one, alert on backend size approaching the quota rather than on the resulting alarm, and remember that compaction alone does not shrink the file - defragmentation does - and that the no-space alarm must be disarmed explicitly before writes resume. When leader elections churn, investigate WAL fsync latency before investigating the network.

Review Questions & Answers

A single cluster-wide counter incremented by every write makes any revision a consistent point-in-time snapshot of the entire store, across all keys. A client can therefore read a range at a revision, then watch from exactly that revision, and be guaranteed to receive every subsequent change with no gap and no duplication. Per-key versions cannot provide that, because there is no way to express a consistent cut across many keys or to resume a multi-key subscription at a known point. This is precisely the property Kubernetes informers depend on: list the current state, note the resourceVersion, and watch onward from it.

etcd retains every historical revision until compaction discards old ones, so a high-churn cluster accumulates history until the backend exceeds its quota, at which point etcd raises a NOSPACE alarm and switches to read-only, allowing reads while refusing writes. Recovery is three steps and each is necessary: compact history to discard revisions older than the current one, defragment each member in turn because compaction frees space logically without shrinking the backend file, and then explicitly disarm the alarm, since it does not clear itself and the cluster stays read-only until it is disarmed. Prevention is configuring auto-compaction retention from the start and alerting on backend size trending toward the quota rather than on the alarm itself.

Because the resourceVersion is an etcd revision, and the error means that revision has been compacted away, so the history needed to replay changes from that point no longer exists anywhere in the cluster. There is nothing to retry: no server can reconstruct events that were deliberately discarded. The client must perform a fresh list to obtain current state and a current revision, then watch from there. This is why informers implement relist-on-expired as a standard path and why aggressive compaction settings combined with slow or frequently disconnected watchers cause a spike in expensive full relists.

Because Raft requires an entry to be durably persisted before it can be acknowledged, so every single commit includes an fsync on the leader and on the followers that acknowledge it, placing storage latency directly on the critical path of every write. If fsync latency approaches or exceeds the election timeout, the leader cannot send heartbeats reliably, followers time out, and leadership churns around the cluster while very little commits. The symptom is intermittent leader elections and write timeouts that look like a network problem, which is why the documented first diagnostic is the WAL fsync and backend commit duration metrics rather than network statistics, and why etcd is expected to run on fast local SSDs rather than network-attached storage.

Events are by far the highest-churn Kubernetes resource, generated continuously by every component for routine occurrences, and each write consumes a consensus round and adds a revision to history that must later be compacted. At scale this write volume can dominate etcd load and drive backend growth, degrading availability for every other object stored in the same cluster. Splitting events out isolates that churn. The general principle is the one from the previous lesson: a coordination service is engineered for a modest volume of very important data, so a single unusually chatty writer can compromise the availability of everything else sharing it, and high-churn data should be isolated or moved out entirely.