Catalog My learning Blog
Apache Kafka Tutorial: Event Streaming to Production · Phase 2 · Lesson 9 · 4 min read

Kafka Replication and In-Sync Replicas (ISR)

On this page
  1. Why Kafka replicates partitions
  2. Partition leaders and followers
  3. The in-sync replica set (ISR)
  4. replication.factor, min.insync.replicas and acks=all
  5. Checking replica state and under-replicated partitions
  6. Unclean leader election
  7. Rack awareness: spreading replicas across zones
  8. Check yourself

Why Kafka replicates partitions

A broker's disk dies. Every partition it held must still exist somewhere, with no acknowledged record lost, and clients must keep working. That's what replication is for, and getting three settings right is most of the job.

Partition leaders and followers

With replication factor 3, each partition exists on three brokers:

  • the leader handles all writes and, by default, all reads,
  • two followers continuously fetch from the leader and append the same records.

If the leader's broker fails, the controller (Day 2) makes one of the followers leader, and clients switch to it automatically.

The in-sync replica set (ISR)

Not every follower is necessarily caught up. The ISR is the set of replicas currently in sync with the leader: the leader itself plus followers that have fetched recently enough (within replica.lag.time.max.ms, 30 seconds by default). A follower that falls behind, for example because its broker is overloaded, is removed from the ISR and rejoins once it catches up.

Only ISR members are eligible to become leader, which is what guarantees a new leader has every acknowledged record.

The same idea protects consumers. The leader tracks a high watermark: the highest offset that every in-sync replica has copied. Consumers can only read up to it. A record the leader has written but followers haven't copied yet stays invisible to consumers, so a leader failover can never take away a record a consumer has already processed.

A partition's leader and two followers; followers that keep up form the in-sync replica set, and acks=all waits for all of them.
A partition's leader and two followers; followers that keep up form the in-sync replica set, and acks=all waits for all of them.

replication.factor, min.insync.replicas and acks=all

  • replication.factor=3 (topic): three copies.
  • min.insync.replicas=2 (topic or broker): a write needs at least 2 in-sync copies to be accepted.
  • acks=all (producer, the default): the producer waits until all current ISR members have the record.

With all three, an acknowledged record exists on at least two brokers. If one broker fails, writes continue (2 replicas left in sync). If two fail, the partition refuses writes with NotEnoughReplicas rather than accept records that exist in only one place. That's a deliberate choice of consistency over availability (CAP, System Design Roadmap Day 5).

Common mistake: acks=all with min.insync.replicas=1. If the ISR shrinks to the leader alone, "all" means one copy, and losing that broker loses acknowledged data.

Checking replica state and under-replicated partitions

Shell
bin/kafka-topics.sh --bootstrap-server broker1:9092 --describe --topic orders
YAML
Topic: orders  Partition: 0  Leader: 2  Replicas: 2,3,1  Isr: 2,3,1
Topic: orders  Partition: 1  Leader: 3  Replicas: 3,1,2  Isr: 3,2

Partition 1's replica on broker 1 has fallen out of sync. That partition is under-replicated: one more failure and it drops below min.insync.replicas. Watch the count of under-replicated partitions across the cluster (Days 23 and 24).

Shell
bin/kafka-topics.sh --bootstrap-server broker1:9092 --describe --under-replicated-partitions

Unclean leader election

If every ISR member is gone, Kafka can either wait for one to return (partition unavailable) or let an out-of-sync replica become leader (unclean.leader.election.enable=true), which brings the partition back but loses the records it was missing. The default is false: prefer unavailability to silent data loss. Enable it only for data where availability matters more than completeness, such as some metrics.

Rack awareness: spreading replicas across zones

Set broker.rack to each broker's availability zone, and Kafka places a partition's replicas in different zones. Losing a whole zone then leaves two copies. Consumers can also read from a nearby follower (client.rack with a rack-aware replica selector) to cut cross-zone traffic costs.

Check yourself

Only one replica remains in sync, below the minimum, so producers with acks=all get errors and writes stop until a replica recovers. Reads can continue.

If the ISR shrinks to the leader alone, an acknowledged write exists on one broker, and losing it loses the data.

It trades data for availability. When no in-sync replica is left, allowing an out-of-sync replica to become leader brings the partition back online, but any records that replica never received are permanently lost. It's off by default, and should stay off for data you can't lose.

With broker.rack set, Kafka spreads each partition's replicas across racks or availability zones, so losing one zone doesn't take out every copy. Consumers can also fetch from a nearby follower to save cross-zone traffic.

The replicas that are caught up with the leader within replica.lag.time.max.ms. Only ISR members can be elected leader cleanly, and acks=all waits for all of them, subject to min.insync.replicas.