Kafka Consumer Offsets and Rebalancing
On this page
Symptom: every deploy causes a lag spike and duplicates
A consumer group of 12 instances is redeployed with a rolling restart. During the deploy, lag jumps, processing stalls for seconds at a time, and some records are processed twice. Each instance leaving and joining triggers a rebalance, and the old rebalance style stopped the whole group every time.
Where committed offsets live: __consumer_offsets
Committed offsets are stored in Kafka itself, in the compacted internal topic __consumer_offsets, keyed by group, topic and partition. When a partition moves to another consumer, the new owner starts from the last committed offset. Anything processed but not yet committed by the previous owner is processed again. That's normal at-least-once behavior (Day 7); rebalances make it happen more often.
What triggers a rebalance
- A consumer joins (scale-up, deploy) or leaves cleanly (shutdown).
- A consumer stops heartbeating for
session.timeout.ms(crash, network). - A consumer doesn't call
poll()withinmax.poll.interval.ms(slow processing). - Partitions are added to a subscribed topic, or a new topic matches a subscription pattern.
Cooperative rebalancing, static membership and KIP-848
Cooperative (incremental) rebalancing. With the old "eager" protocol, every member gave up all its partitions and waited for a new assignment. With the CooperativeStickyAssignor (KIP-429), members keep the partitions they're allowed to keep, and only the ones that move are paused.
Static membership. Give each instance a stable group.instance.id (for example its Pod name). A restarting member with the same id gets its old partitions back without a rebalance, as long as it returns within session.timeout.ms. Ideal for rolling deploys.
The new consumer group protocol (KIP-848). Production-ready in Kafka 4.x and enabled with group.protocol=consumer: assignment moves to the broker, rebalances become incremental by design, and a slow member no longer blocks the whole group. For new deployments on Kafka 4.x, it's worth adopting.
Commit on revoke. When partitions are being taken away, commit their offsets first, so the next owner reprocesses as little as possible. Client libraries expose callbacks (on_revoke, ConsumerRebalanceListener) for this.
Fixing rebalances during rolling deploys
consumer = Consumer({
"bootstrap.servers": "kafka:9092",
"group.id": "order-processors",
"group.instance.id": os.environ["POD_NAME"], # static membership
"partition.assignment.strategy": "cooperative-sticky",
"session.timeout.ms": 45000,
"enable.auto.commit": False,
})With the instance id stable across restarts and a session timeout longer than a Pod restart, rolling deploys cause few or no rebalances.
How to reset consumer group offsets
Sometimes you need to reprocess (after fixing a bug) or skip ahead (after an incident). Stop the group's consumers first; offsets can only be reset for an inactive group.
bin/kafka-consumer-groups.sh --bootstrap-server kafka:9092 --group order-processors \
--topic orders --reset-offsets --to-datetime 2026-10-09T00:00:00.000 --dry-run
# review the planned offsets, then replace --dry-run with --executeOther targets: --to-earliest, --to-latest, --shift-by -1000, --to-offset. Always --dry-run first.
Check yourself
In Kafka's internal topic __consumer_offsets, which is compacted and keyed by group, topic and partition. Storing offsets in Kafka itself means they're replicated like any other data and survive consumer and broker restarts.
A restarting consumer with the same group.instance.id rejoins and keeps its partitions without triggering a rebalance, if it returns within the session timeout.
Stop the group's consumers, then run kafka-consumer-groups.sh --reset-offsets --to-datetime ... with --dry-run first to check the positions, then --execute. Restart the consumers to reprocess from there. Because records will be processed again, the processing must be idempotent.
With eager rebalancing every consumer gives up all its partitions and processing stops. Cooperative rebalancing moves only the partitions that need to move, so the rest keep processing, which greatly reduces pauses during deploys and scaling.
A new consumer group protocol in which the broker computes partition assignments and rebalances happen incrementally per member, removing global synchronization barriers. It makes rebalances faster and less disruptive.