Catalog My learning Blog
Apache Kafka Tutorial: Event Streaming to Production · Phase 1 · Lesson 6 · 4 min read

Kafka Consumers and Consumer Groups Explained

On this page
  1. Scaling a slow Kafka consumer
  2. How consumer groups divide partitions
  3. Multiple consumer groups: each gets every record
  4. Heartbeats, max.poll.interval.ms and rebalances
  5. Consumer lag: the health signal
  6. Scaling options when lag grows
  7. Parallel processing inside one consumer
  8. Check yourself

Scaling a slow Kafka consumer

One consumer processes the 6-partition orders topic, and its lag keeps growing. Start a second copy with the same group.id and the work splits. Start a third, a sixth: each new member takes some partitions. Start a seventh, and it sits idle. That's the consumer group model in one paragraph.

How consumer groups divide partitions

Consumers that share a group.id form a consumer group. Kafka assigns each partition to exactly one consumer in the group at a time:

Consumers in the groupPartitions each (6-partition topic)
16
23
32
61
86 consumers get 1; 2 sit idle

Because a partition has one owner in a group, records in that partition are processed in order. The partition count is the ceiling on parallelism within one group; plan it from your target throughput (Day 22).

Within one consumer group, each partition is assigned to one consumer; separate groups each read every partition independently.
Within one consumer group, each partition is assigned to one consumer; separate groups each read every partition independently.

Multiple consumer groups: each gets every record

Groups don't share work across each other. The billing group and the analytics group each receive every record, each with its own committed offsets and its own pace. That's how Kafka does publish/subscribe: one topic, many independent readers.

Heartbeats, max.poll.interval.ms and rebalances

A group coordinator, a broker, tracks each group's members. Consumers send heartbeats; the coordinator removes a member that stops heartbeating (session.timeout.ms, 45 seconds by default) or that takes too long between poll() calls (max.poll.interval.ms, 5 minutes by default). When members join or leave, partitions are reassigned: a rebalance.

The second limit catches a common bug: processing a batch that takes longer than max.poll.interval.ms. The consumer is evicted mid-batch, its partitions move to another member, and the same records get processed twice. Process faster, fetch fewer records per poll (max.poll.records), or raise the limit deliberately.

Rebalances used to pause the whole group ("stop the world"). Two improvements:

  • Cooperative rebalancing (the CooperativeStickyAssignor): only the partitions that need to move are paused.
  • The new consumer group protocol (KIP-848), production-ready in Kafka 4.x and enabled per consumer with group.protocol=consumer: the broker computes assignments incrementally, making rebalances much less disruptive. Kafka 4.3's console tools even print a reminder that it's available.

Day 11 goes deeper into offsets and rebalancing.

Consumer lag: the health signal

Shell
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group order-processors
Code
GROUP             TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG   CONSUMER-ID
order-processors  orders  0          10522           10523           1     consumer-1-...
order-processors  orders  1          9310            24890           15580 consumer-2-...

Lag is how many records the group still has to process. Steady small lag is normal. Growing lag means consumers can't keep up; lag concentrated on one partition means a hot key or a stuck consumer. Alert on lag, or better, on how long the lag would take to clear (Day 23).

Scaling options when lag grows

  1. Add consumers to the group, up to the partition count.
  2. Make processing faster: batch database writes, avoid synchronous calls per record.
  3. Add partitions, carefully, because it remaps keys (Day 3).
  4. Within one consumer, process partitions in parallel while keeping per-key order.

Parallel processing inside one consumer

Option 4 needs care, because two things break easily. Handing records to a thread pool loses their order, and committing offset N tells Kafka that every record before N is finished. The safe shape:

  • Route each record to a worker by key, hash(key) % workers, so one key's records stay on one worker, in order.
  • Track, per partition, which offsets have finished. You can only commit up to the lowest offset still in progress, so one slow record holds back the commit, and after a crash everything from that offset on is processed again.
  • Bound the number of records in flight, or a slow downstream fills memory.

This pays off when partitions are few and each record waits on I/O, such as an API call: one consumer can then work on hundreds of keys at once instead of one record per partition. Confluent's Parallel Consumer library implements exactly this for Java, with per-key ordering and offset tracking built in.

Check yourself

Six consumers each get one partition and four sit idle, ready to take over if one fails. Within a consumer group, each partition goes to exactly one consumer, so the partition count caps the group's parallelism.

Give them different consumer group ids. Each group independently receives every record in the topic and keeps its own committed offsets, so the two services read at their own pace without affecting each other.

The time between its calls to poll() exceeds max.poll.interval.ms, so the group coordinator assumes it's stuck and triggers a rebalance. Fetch fewer records per poll with max.poll.records, make processing faster, or raise the interval to match the real processing time.

Route records to workers by key so each key's records stay on one worker in order, and commit only up to the lowest offset that's still in progress.