Kafka Architecture: Brokers, ZooKeeper vs KRaft
On this page
Kafka brokers and clusters
A broker is a Kafka server process. It stores partitions of topics on its local disks, accepts writes from producers and serves reads to consumers. A cluster is several brokers working together, typically three or more, so data can be replicated and the cluster survives a broker failure.
Clients only need one or two broker addresses to start (the bootstrap servers). From any broker they fetch the cluster metadata: which topics exist, how they're partitioned, and which broker currently leads each partition. They then talk directly to the right brokers.
The Kafka controller and cluster metadata
Someone must keep track of cluster state: which brokers are alive, which broker leads each partition, which replicas are in sync, topic configurations. That's the controller. When a broker dies, the controller elects new leaders for its partitions and tells everyone.
ZooKeeper vs KRaft: why Kafka dropped ZooKeeper
For most of Kafka's history, metadata lived in Apache ZooKeeper, a separate coordination system. That meant operating two distributed systems, and it limited scale: failover after a controller change had to reload metadata from ZooKeeper, which got slow with hundreds of thousands of partitions.
KRaft (Kafka Raft) moves metadata into Kafka itself. A small group of controller nodes stores metadata as an internal, replicated log using a Raft-based consensus protocol (the same idea as etcd's, covered in the System Design Roadmap). Brokers follow that log. Benefits: one system to run, faster failover, and support for far more partitions.
The timeline: KRaft became production-ready in Kafka 3.3, ZooKeeper mode was deprecated in 3.5, and Kafka 4.0 removed ZooKeeper entirely. Every new cluster is KRaft. Older clusters had to migrate on 3.x before upgrading to 4.x.
KRaft process roles: broker and controller
Each node's process.roles setting decides what it does:
| Role | Does |
|---|---|
broker | Stores data and serves clients |
controller | Votes in the metadata quorum |
broker,controller | Both: fine for development and small clusters |
Production clusters usually run 3 dedicated controllers (tolerating one failure; 5 tolerate two) and as many brokers as the load requires. Controllers need little disk or CPU but must be reliable, so keep them on separate machines or availability zones.
Formatting KRaft storage
Before a KRaft node starts the first time, its storage is formatted with a cluster id. For a single-node development setup, as used in this course:
KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
bin/kafka-storage.sh format --standalone -t "$KAFKA_CLUSTER_ID" -c config/server.properties
bin/kafka-server-start.sh config/server.propertiesOn Kafka 4.3, the format step prints Formatting dynamic metadata voter directory ... with metadata.version 4.3-IV0. --standalone creates a one-node controller quorum. In a multi-node cluster you format every node with the same cluster id and the list of initial controllers; a node formatted with a different id refuses to join, which protects against mixing clusters.
Check yourself
The controller manages the cluster's metadata: which brokers are alive, which broker leads each partition, the in-sync replica sets and topic configurations. When a broker fails, it elects new leaders for that broker's partitions and propagates the change. In KRaft mode, a quorum of controllers stores this metadata in its own replicated log.
To run one system instead of two, speed up controller failover, and support many more partitions by keeping metadata in Kafka's own replicated log.
The quorum needs a majority to work; 3 nodes tolerate one failure and 5 tolerate two, while even numbers add cost without extra tolerance.
kafka-storage.sh format writes the cluster id and metadata version into each node's log directories, so every broker and controller agrees which cluster it belongs to. A node with a mismatched or missing cluster id refuses to start, which prevents accidentally joining the wrong cluster.
Brokers store partitions and serve producers and consumers. Controllers form the metadata quorum and make cluster decisions. Small or development clusters can run both roles in one process; production clusters usually run dedicated controllers so metadata work isn't affected by data traffic.