Kafka Consumers and Consumer Groups Explained
On this page
Scaling a slow Kafka consumer
One consumer processes the 6-partition orders topic, and its lag keeps growing. Start a second copy with the same group.id and the work splits. Start a third, a sixth: each new member takes some partitions. Start a seventh, and it sits idle. That's the consumer group model in one paragraph.
How consumer groups divide partitions
Consumers that share a group.id form a consumer group. Kafka assigns each partition to exactly one consumer in the group at a time:
| Consumers in the group | Partitions each (6-partition topic) |
|---|---|
| 1 | 6 |
| 2 | 3 |
| 3 | 2 |
| 6 | 1 |
| 8 | 6 consumers get 1; 2 sit idle |
Because a partition has one owner in a group, records in that partition are processed in order. The partition count is the ceiling on parallelism within one group; plan it from your target throughput (Day 22).
Multiple consumer groups: each gets every record
Groups don't share work across each other. The billing group and the analytics group each receive every record, each with its own committed offsets and its own pace. That's how Kafka does publish/subscribe: one topic, many independent readers.
Heartbeats, max.poll.interval.ms and rebalances
A group coordinator, a broker, tracks each group's members. Consumers send heartbeats; the coordinator removes a member that stops heartbeating (session.timeout.ms, 45 seconds by default) or that takes too long between poll() calls (max.poll.interval.ms, 5 minutes by default). When members join or leave, partitions are reassigned: a rebalance.
The second limit catches a common bug: processing a batch that takes longer than max.poll.interval.ms. The consumer is evicted mid-batch, its partitions move to another member, and the same records get processed twice. Process faster, fetch fewer records per poll (max.poll.records), or raise the limit deliberately.
Rebalances used to pause the whole group ("stop the world"). Two improvements:
- Cooperative rebalancing (the
CooperativeStickyAssignor): only the partitions that need to move are paused. - The new consumer group protocol (KIP-848), production-ready in Kafka 4.x and enabled per consumer with
group.protocol=consumer: the broker computes assignments incrementally, making rebalances much less disruptive. Kafka 4.3's console tools even print a reminder that it's available.
Day 11 goes deeper into offsets and rebalancing.
Consumer lag: the health signal
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group order-processorsGROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID
order-processors orders 0 10522 10523 1 consumer-1-...
order-processors orders 1 9310 24890 15580 consumer-2-...Lag is how many records the group still has to process. Steady small lag is normal. Growing lag means consumers can't keep up; lag concentrated on one partition means a hot key or a stuck consumer. Alert on lag, or better, on how long the lag would take to clear (Day 23).
Scaling options when lag grows
- Add consumers to the group, up to the partition count.
- Make processing faster: batch database writes, avoid synchronous calls per record.
- Add partitions, carefully, because it remaps keys (Day 3).
- Within one consumer, process partitions in parallel while keeping per-key order.
Parallel processing inside one consumer
Option 4 needs care, because two things break easily. Handing records to a thread pool loses their order, and committing offset N tells Kafka that every record before N is finished. The safe shape:
- Route each record to a worker by key,
hash(key) % workers, so one key's records stay on one worker, in order. - Track, per partition, which offsets have finished. You can only commit up to the lowest offset still in progress, so one slow record holds back the commit, and after a crash everything from that offset on is processed again.
- Bound the number of records in flight, or a slow downstream fills memory.
This pays off when partitions are few and each record waits on I/O, such as an API call: one consumer can then work on hundreds of keys at once instead of one record per partition. Confluent's Parallel Consumer library implements exactly this for Java, with per-key ordering and offset tracking built in.
Check yourself
Six consumers each get one partition and four sit idle, ready to take over if one fails. Within a consumer group, each partition goes to exactly one consumer, so the partition count caps the group's parallelism.
Give them different consumer group ids. Each group independently receives every record in the topic and keeps its own committed offsets, so the two services read at their own pace without affecting each other.
The time between its calls to poll() exceeds max.poll.interval.ms, so the group coordinator assumes it's stuck and triggers a rebalance. Fetch fewer records per poll with max.poll.records, make processing faster, or raise the interval to match the real processing time.
Route records to workers by key so each key's records stay on one worker in order, and commit only up to the lowest offset that's still in progress.