Catalog My learning Blog
Apache Kafka Tutorial: Event Streaming to Production · Phase 4 · Lesson 24 · 3 min read

Kafka Broker Failures and Under-Replicated Partitions

On this page
  1. What happens when a Kafka broker dies
  2. Finding under-replicated and offline partitions
  3. When the broker comes back
  4. Broker failure runbook
  5. Practice broker failures in staging
  6. Check yourself

What happens when a Kafka broker dies

Broker 3 in a five-broker cluster loses power. With replication factor 3 and min.insync.replicas=2 (Day 9):

  1. The controller notices broker 3 has stopped heartbeating (seconds, set by broker.session.timeout.ms).
  2. For every partition broker 3 led, the controller picks a new leader from that partition's ISR and updates the metadata.
  3. Producers and consumers get errors such as NOT_LEADER_OR_FOLLOWER, refresh metadata, and continue with the new leaders. With retries and idempotence on (Day 5), producers don't lose or duplicate records.
  4. Partitions where broker 3 was a follower keep their leaders but now have only two in-sync replicas: they're under-replicated, but still accept writes because 2 meets the minimum.

The cluster keeps working. Nothing is lost. But the safety margin is gone for those partitions: one more failure among their replicas and writes stop.

When a broker fails, its partitions' leadership moves to in-sync followers; partitions it followed become under-replicated until it returns and catches up.
When a broker fails, its partitions' leadership moves to in-sync followers; partitions it followed become under-replicated until it returns and catches up.

Finding under-replicated and offline partitions

Shell
bin/kafka-topics.sh --bootstrap-server kafka1:9092 --describe --under-replicated-partitions
bin/kafka-topics.sh --bootstrap-server kafka1:9092 --describe --under-min-isr-partitions
bin/kafka-topics.sh --bootstrap-server kafka1:9092 --describe --unavailable-partitions
YAML
Topic: orders  Partition: 4  Leader: 1  Replicas: 1,3,5  Isr: 1,5
Topic: orders  Partition: 9  Leader: 5  Replicas: 3,5,2  Isr: 5,2

Broker 3 is missing from every ISR it should be in: that confirms which broker the problem is.

When the broker comes back

The returning broker's replicas aren't in sync yet. Each fetches everything it missed from the current leaders and rejoins the ISR once caught up. That can take minutes to hours for a broker that held a lot of data (tiered storage, Day 22, shortens it). Leadership doesn't automatically jump back immediately; with auto.leader.rebalance.enable=true (the default), the controller periodically moves leadership back to each partition's preferred leader so load evens out again.

Broker failure runbook

Under-replicated partitions, one broker absent from many ISRs:

  1. Check that broker: process up? disk full or failing? network? JVM pauses in the logs?
  2. If it's down, restart it and watch URP fall as it catches up.
  3. If the hardware is gone, replace the broker with the same broker id and empty storage; it re-replicates its partitions from leaders.

Under-replicated partitions spread across many brokers:

  • Usually overload, not failure: disks or network saturated, followers can't keep up. Check throughput, request handler idle ratio, and recent partition reassignments that may be copying data.

Under-min-ISR or offline partitions:

  • Writes (or everything) are failing for those partitions. Restore a replica's broker first. Unclean leader election (Day 9) brings an offline partition back at the cost of data loss; use it only with a deliberate decision.

Moving data off a broker (decommissioning, rebalancing): use kafka-reassign-partitions.sh with a generated plan and a throttle (--throttle) so replication traffic doesn't starve clients, or a tool such as Cruise Control.

Practice broker failures in staging

Kill a broker in a staging cluster under realistic load and watch: how long until clients recover, how long URP lasts, whether any producer errors reach users. Do it before production does it for you (System Design Roadmap, Day 79).

Check yourself

Yes. For partitions the failed broker led, the controller elects new leaders from the remaining in-sync replicas. Every partition still has two in-sync copies, which meets min.insync.replicas=2, so acks=all writes keep succeeding. A second failure would block writes to affected partitions.

Overload rather than failure: followers can't replicate fast enough because disks, network or request handler threads are saturated, or a large partition reassignment is consuming bandwidth. Check broker resource metrics and any reassignment in progress.

Start a new broker with the same broker id and empty storage; it re-replicates its partitions from the current leaders and rejoins the ISRs.

It rejoins, its replicas catch up from the current leaders, and once in sync they rejoin the ISR. Leadership then moves back gradually through preferred leader election, which balances load again.