Catalog My learning Blog
Apache Kafka Tutorial: Event Streaming to Production · Phase 2 · Lesson 11 · 3 min read

Kafka Consumer Offsets and Rebalancing

On this page
  1. Symptom: every deploy causes a lag spike and duplicates
  2. Where committed offsets live: __consumer_offsets
  3. What triggers a rebalance
  4. Cooperative rebalancing, static membership and KIP-848
  5. Fixing rebalances during rolling deploys
  6. How to reset consumer group offsets
  7. Check yourself

Symptom: every deploy causes a lag spike and duplicates

A consumer group of 12 instances is redeployed with a rolling restart. During the deploy, lag jumps, processing stalls for seconds at a time, and some records are processed twice. Each instance leaving and joining triggers a rebalance, and the old rebalance style stopped the whole group every time.

Where committed offsets live: __consumer_offsets

Committed offsets are stored in Kafka itself, in the compacted internal topic __consumer_offsets, keyed by group, topic and partition. When a partition moves to another consumer, the new owner starts from the last committed offset. Anything processed but not yet committed by the previous owner is processed again. That's normal at-least-once behavior (Day 7); rebalances make it happen more often.

What triggers a rebalance

  • A consumer joins (scale-up, deploy) or leaves cleanly (shutdown).
  • A consumer stops heartbeating for session.timeout.ms (crash, network).
  • A consumer doesn't call poll() within max.poll.interval.ms (slow processing).
  • Partitions are added to a subscribed topic, or a new topic matches a subscription pattern.
Members join and leave a consumer group; cooperative rebalancing moves only the partitions that must change, instead of pausing everyone.
Members join and leave a consumer group; cooperative rebalancing moves only the partitions that must change, instead of pausing everyone.

Cooperative rebalancing, static membership and KIP-848

Cooperative (incremental) rebalancing. With the old "eager" protocol, every member gave up all its partitions and waited for a new assignment. With the CooperativeStickyAssignor (KIP-429), members keep the partitions they're allowed to keep, and only the ones that move are paused.

Static membership. Give each instance a stable group.instance.id (for example its Pod name). A restarting member with the same id gets its old partitions back without a rebalance, as long as it returns within session.timeout.ms. Ideal for rolling deploys.

The new consumer group protocol (KIP-848). Production-ready in Kafka 4.x and enabled with group.protocol=consumer: assignment moves to the broker, rebalances become incremental by design, and a slow member no longer blocks the whole group. For new deployments on Kafka 4.x, it's worth adopting.

Commit on revoke. When partitions are being taken away, commit their offsets first, so the next owner reprocesses as little as possible. Client libraries expose callbacks (on_revoke, ConsumerRebalanceListener) for this.

Fixing rebalances during rolling deploys

Code
consumer = Consumer({
    "bootstrap.servers": "kafka:9092",
    "group.id": "order-processors",
    "group.instance.id": os.environ["POD_NAME"],   # static membership
    "partition.assignment.strategy": "cooperative-sticky",
    "session.timeout.ms": 45000,
    "enable.auto.commit": False,
})

With the instance id stable across restarts and a session timeout longer than a Pod restart, rolling deploys cause few or no rebalances.

How to reset consumer group offsets

Sometimes you need to reprocess (after fixing a bug) or skip ahead (after an incident). Stop the group's consumers first; offsets can only be reset for an inactive group.

Shell
bin/kafka-consumer-groups.sh --bootstrap-server kafka:9092 --group order-processors \
  --topic orders --reset-offsets --to-datetime 2026-10-09T00:00:00.000 --dry-run
# review the planned offsets, then replace --dry-run with --execute

Other targets: --to-earliest, --to-latest, --shift-by -1000, --to-offset. Always --dry-run first.

Check yourself

In Kafka's internal topic __consumer_offsets, which is compacted and keyed by group, topic and partition. Storing offsets in Kafka itself means they're replicated like any other data and survive consumer and broker restarts.

A restarting consumer with the same group.instance.id rejoins and keeps its partitions without triggering a rebalance, if it returns within the session timeout.

Stop the group's consumers, then run kafka-consumer-groups.sh --reset-offsets --to-datetime ... with --dry-run first to check the positions, then --execute. Restart the consumers to reprocess from there. Because records will be processed again, the processing must be idempotent.

With eager rebalancing every consumer gives up all its partitions and processing stops. Cooperative rebalancing moves only the partitions that need to move, so the rest keep processing, which greatly reduces pauses during deploys and scaling.

A new consumer group protocol in which the broker computes partition assignments and rebalances happen incrementally per member, removing global synchronization barriers. It makes rebalances faster and less disruptive.