Catalog My learning Blog
Raft and Paxos Explained: Consensus, ZooKeeper and etcd · Phase 3 · Lesson 16 · 11 min read

Coordination Service vs Embedded Raft

On this page
  1. Learning Objective
  2. Concept Explanation
  3. Worked Example: Three Requirements, Three Different Answers
  4. Real-World Examples
  5. Review Questions & Answers

Learning Objective

Decide between depending on an external coordination service and embedding a consensus library in your own application, using the size and churn of the agreed state, the failure-domain implications, and the operational cost of each as the deciding criteria.

Concept Explanation

Once a system needs agreement, there are three real options rather than two. Depend on an external coordination service such as ZooKeeper, etcd, or Consul. Embed a consensus library - etcd's raft package, HashiCorp's raft library, Apache Ratis - directly in your own process so your application is itself a Raft group. Or restructure the problem so you do not need distributed agreement at all, typically by pushing the requirement into a database you already run that provides the necessary atomicity. The third option is chosen less often than it should be, and it is worth considering first precisely because it is the cheapest to operate.

The external coordination service is the right default when the agreed state is small, changes infrequently, and is about the system rather than the data: who is the leader, which nodes are members, what is the current configuration, which shard is assigned where. You get a mature, heavily tested consensus implementation that you do not have to debug, an operational model your team can learn once and reuse, and a failure domain separate from your application, so an application bug cannot corrupt the coordination state. The costs are equally concrete. You have another distributed system to run, upgrade, back up, and monitor. You add a network round trip to operations that touch it. Most significantly, you create an availability dependency: your application is now at most as available as the coordination service, which is why teams that use one for anything on the request path end up caching aggressively and designing explicit degraded modes for when it is unreachable.

Embedding consensus is the right answer when agreement is about the data itself and must scale with it. The decisive question is whether one consensus group can carry the whole workload. If every write in the system would have to pass through a single leader, an external service becomes a throughput ceiling, and the standard answer is many consensus groups: CockroachDB runs a Raft group per range of roughly 512 MiB, TiKV runs one per region, and both operate thousands of groups concurrently so unrelated writes commit in parallel while each key range still gets a total order. No external coordination service can offer that, because its entire design point is a single small replicated state. Embedding also removes the extra network hop and the external availability dependency. What it costs is real and frequently underestimated: you now own a consensus implementation's failure modes, its upgrade and membership-change paths, its snapshotting behaviour, and the testing burden that goes with correctness-critical distributed code.

A short set of questions usually settles the decision. How large is the agreed state? Kilobytes to a few megabytes points to an external service; gigabytes and growing points to embedding. How often does it change? A handful of writes per second points to a service; thousands per second per shard points to embedding. Is it on the request path? If every user request must consult it, the added hop and the availability coupling argue for caching with a degraded mode, or for embedding. Does it need to scale horizontally? A single consensus group cannot, and if the answer is yes then you need many groups, which means embedding. And finally, does your team have the operational capacity? Running ZooKeeper or etcd well is a real skill; building and testing a correct embedded consensus layer is a considerably larger one.

Two anti-patterns are worth naming explicitly because both are common. The first is using a coordination service as a general-purpose database, storing per-user session state, job queues, or metrics, which drives the write churn and data volume it was never designed for and ends in a quota alarm that freezes everything else depending on it. The second is building a bespoke consensus implementation from a paper because the existing libraries seemed heavyweight. Consensus code is correctness-critical, its bugs are silent, and they surface only under the precise failure combinations that are hardest to reproduce; the published implementations have been exercised by years of production traffic and by formal verification and fault-injection testing that a new implementation will not match. If consensus must be embedded, embed a library rather than an implementation of your own.

Deciding FactorPoints to an External ServicePoints to Embedded Consensus
Size of agreed stateKilobytes to a few megabytesGigabytes, growing with the dataset
Write rate to agreed stateTens to hundreds per secondThousands per second, per shard
Scaling shapeOne small state the whole cluster sharesMany independent groups, one per shard or range
Request-path involvementConsulted rarely, or cached with a degraded modeOn the critical path of every data write
Team capacityCan operate one more distributed systemCan own consensus failure modes and testing
Representative systemsHadoop, HBase, Solr, Kubernetes componentsCockroachDB, TiKV, Kafka KRaft, Consul itself

Worked Example: Three Requirements, Three Different Answers

The same team can correctly reach three different conclusions for three features in the same product, which is the clearest demonstration that this is a per-requirement decision rather than an architectural preference.

SQL
# REQUIREMENT 1: "Exactly one instance of our batch service should run
#                 the nightly reconciliation job."
#
# Agreed state:   one key naming the current leader
# Size:           a few dozen bytes
# Write rate:     one change per leadership handover, maybe daily
# Request path:   no
# -> EXTERNAL COORDINATION SERVICE
lease = etcd.lease_grant(ttl=15)
etcd.txn(compare=[Compare('/jobs/nightly/leader', 'create_revision', 0)],
         success=[Put('/jobs/nightly/leader', instance_id, lease=lease)])
# Plus: make the job itself idempotent, so a pause-window overlap is
# harmless without needing fencing tokens on every downstream write.


# REQUIREMENT 2: "Our database must keep three consistent replicas of
#                 40 TB of data and survive a node failure."
#
# Agreed state:   the data itself
# Size:           40 TB
# Write rate:     hundreds of thousands per second, cluster-wide
# Scaling shape:  must scale horizontally; one leader cannot carry it
# -> EMBEDDED CONSENSUS, MANY GROUPS (what CockroachDB and TiKV do)
#    Split the key space into ranges; one Raft group per range; each
#    group has its own leader, log, and quorum. Unrelated writes commit
#    in parallel while each range still gets a total order.
#    An external coordination service cannot do this: its design point
#    is ONE small replicated state.


# REQUIREMENT 3: "Two workers must not process the same order twice."
#
# The instinct is a distributed lock. Check first whether agreement is
# needed at all - the orders table is already transactional.
# -> NO DISTRIBUTED CONSENSUS AT ALL
UPDATE orders
   SET status = 'processing', worker_id = $1, claimed_at = now()
 WHERE id = $2 AND status = 'pending';
-- 0 rows affected means another worker claimed it first.

# One database round trip. No lock service, no lease, no fencing token,
# no new availability dependency, and no pause window: the exclusion is
# enforced by the same transaction that does the work, so a paused
# worker cannot cause a double claim. Strictly better than a lock here.


# THE RULE OF THUMB
# Ask, in this order:
#   1. Can an existing transactional store provide this atomically?
#      -> use it. Cheapest to operate, fewest failure modes.
#   2. Is the agreed state small, rarely changed, and about the SYSTEM?
#      -> external coordination service.
#   3. Is the agreed state the DATA, and must it scale horizontally?
#      -> embed a consensus LIBRARY, many groups. Never write your own.
A decision path across three architectures: an existing transactional store providing atomicity directly, an external coordination service holding one small shared state, and embedded consensus running many independent groups that scale with the data.
A decision path across three architectures: an existing transactional store providing atomicity directly, an external coordination service holding one small shared state, and embedded consensus running many independent groups that scale with the data.

Real-World Examples

  • Apache Kafka moved in the opposite direction from most systems, replacing its external ZooKeeper dependency with KRaft, an embedded Raft implementation, which became the default in Kafka 3.3 and the only option in Kafka 4.0. The stated motivations were that metadata for very large clusters had outgrown ZooKeeper's design point of a small in-memory dataset, that operating two distributed systems doubled the operational burden, and that an embedded metadata log removed a network hop and a separate failure domain from the controller path.
  • CockroachDB and TiKV both embed Raft and run thousands of independent groups, one per data range or region, precisely because their agreed state is the data itself and must scale horizontally. Neither could have been built on an external coordination service, since every write in the system would have had to pass through a single consensus leader, and the per-range design is what allows unrelated writes across the key space to commit concurrently while each individual range still receives a total order.
Best PracticeAsk the questions in order: can a transactional store you already run provide this atomically, is the agreed state small and rarely changed and about the system rather than the data, or must agreement scale horizontally with the data? Those three map cleanly to a conditional write, an external coordination service, and embedded consensus with many groups. And if you embed, embed a proven library - consensus bugs are silent, surface only under the failure combinations hardest to reproduce, and are not worth rediscovering.

Review Questions & Answers

That it must scale horizontally with the data. A coordination service is architecturally one small replicated state behind a single consensus group, so every write passes through one leader and total throughput is bounded no matter how the service is deployed. If the requirement is to keep terabytes consistent across replicas at high write rates, that ceiling is fatal, and the answer is many independent consensus groups partitioned across the key space, which requires embedding. Size and write rate are useful secondary signals, but scaling shape is the one that cannot be worked around by tuning.

Because your application immediately becomes at most as available as the coordination service, and a service that is down or has lost quorum will stall every request that consults it. Consensus systems deliberately choose unavailability over inconsistency, so the failure is not rare or theoretical: any loss of quorum, any maintenance error, any backend quota alarm produces exactly this. A design that puts one on the request path therefore needs a deliberate answer to what happens when it is unreachable, usually caching the last known value with a bounded staleness and a documented behaviour for operating on stale coordination data, rather than discovering the answer during an incident.

A single group would serialise every write in the system behind one leader, making the cluster's write throughput independent of how many machines it contains and turning the leader into an unavoidable bottleneck. Partitioning the key space into ranges and giving each range its own Raft group with its own leader and quorum means writes to unrelated ranges commit fully in parallel across different leaders on different machines, so throughput scales with the cluster. Each individual range still receives a total order and full consensus guarantees, so nothing is given up in correctness - only the unnecessary global ordering between unrelated keys.

Session state is high-churn and grows with the user base, which is the opposite of what a coordination service is designed for. Every session write becomes a consensus round replicated to every ensemble member and held in memory on each, so throughput is capped in the low tens of thousands of writes per second and memory pressure rises with active users. Worse, the failure is shared: when the ensemble degrades under this load or hits a size limit, every other consumer of it - leader elections, membership, configuration - degrades with it. Session state belongs in a store designed for high write volume and partitioning, with the coordination service reserved for the small facts the whole cluster must agree on.

Kafka's agreed state is cluster metadata - broker registrations, topic and partition configuration, and leadership assignments - which for very large clusters had grown beyond ZooKeeper's design point of a small in-memory dataset, and the propagation path required brokers to poll and reconcile external state rather than replay an ordered log. Embedding Raft turned metadata into a replicated log every broker replays, giving each broker an identical resumable position, removing a network hop and a separate failure domain from the controller path, and eliminating the burden of operating and upgrading a second distributed system. The general lesson is that the decision follows from the size, churn, and shape of the agreed state, and can legitimately change for the same system as it grows.