Catalog My learning Blog
Apache Kafka Tutorial: Event Streaming to Production · Phase 4 · Lesson 27 · 4 min read

Kafka in Production: Pitfalls and Runbook

On this page
  1. Before go-live: reviewing a Kafka service
  2. Kafka pre-production checklist
  3. Common Kafka pitfalls and fixes
  4. Kafka incident runbook
  5. Check yourself

Before go-live: reviewing a Kafka service

A new service is about to start producing to and consuming from Kafka in production. Review it against the checklist below before go-live, then make sure on-call has the runbook. Most serious Kafka incidents trace back to one of these items.

Kafka pre-production checklist

Topics

  • [ ] Created deliberately, not auto-created; auto-creation disabled on the cluster (Day 4).
  • [ ] Partition count planned from measured throughput and consumer parallelism, with headroom (Day 22).
  • [ ] Replication factor 3, min.insync.replicas=2, replicas spread across zones (Day 9).
  • [ ] Retention or compaction chosen for the data's purpose (Day 10).
  • [ ] Schema registered with a compatibility rule (Day 8).

Producers

  • [ ] acks=all and idempotence on (the defaults; check nobody overrode them) (Day 5).
  • [ ] Keys chosen for the ordering that's needed, and the same partitioner across client libraries (Days 5 and 14).
  • [ ] Delivery failures handled in callbacks, flush() on shutdown.
  • [ ] Events from database changes published through an outbox, not dual writes (Day 19).

Consumers

  • [ ] Auto-commit off for anything important; commit after processing (Day 7).
  • [ ] Processing idempotent; duplicates are expected, not exceptional.
  • [ ] Poison messages go to a dead-letter topic instead of blocking a partition.
  • [ ] max.poll.interval.ms comfortably above the slowest batch; static membership or the new group protocol for rolling deploys (Day 11).

Cluster and operations

  • [ ] TLS, authentication and least-privilege ACLs per service (Day 25).
  • [ ] Alerts: offline partitions, under-min-ISR, URP, controller count, disk, time-based consumer lag (Day 23).
  • [ ] Broker failure and DR failover tested in staging (Days 24 and 26).
The most common Kafka pitfalls mapped to their fixes: planned partitions, durable producers, idempotent consumers with dead-letter topics, security, monitoring and tested failover.
The most common Kafka pitfalls mapped to their fixes: planned partitions, durable producers, idempotent consumers with dead-letter topics, security, monitoring and tested failover.

Common Kafka pitfalls and fixes

PitfallWhat happensFix
Auto-commit with side effectsLost or duplicated processing after crashesManual commit after processing, idempotent handlers
min.insync.replicas=1 with acks=allAcknowledged data lost when one broker diesmin.insync.replicas=2
One poison messageA partition stops; lag grows foreverDead-letter topic and skip after N attempts
Changing partition count on keyed topicsPer-key ordering breaksPlan partitions up front, or migrate to a new topic
Treating Kafka as permanent storage without planningData silently deleted by retentionExplicit retention, tiered storage, or a lake for history
No lag alertingConsumers hours behind before anyone noticesTime-based lag alerts per critical group
Never testing failoverDR fails on a detail during a real outageRegular game days

Kafka incident runbook

Consumer lag growing

  1. Is lag on all partitions or one? One: hot key or a stuck instance (check its logs and the key distribution). All: the group is too slow.
  2. Check for frequent rebalances (instances crashing or exceeding max.poll.interval.ms).
  3. Check the downstream: a slow database or API usually explains slow consumers.
  4. Scale consumers up to the partition count; optimize processing; batch writes.

Producer errors or timeouts

  1. Check for under-min-ISR or offline partitions on the target topic.
  2. Check broker request latency and disk.
  3. Check client-side buffer exhaustion and authorization errors.

Under-replicated partitions (Day 24): find the common broker; check health, disk and network; restart or replace; throttle any reassignment.

Disk filling up

  1. Find the largest topics (kafka-log-dirs.sh --describe).
  2. Lower retention temporarily on the offending topic, or add brokers/storage and rebalance.
  3. Find the cause: a producer writing far more than planned, or a compacted topic that isn't compacting.

Poison message blocking a partition

  1. Find the offset from consumer logs.
  2. Fix the consumer to dead-letter it, or, as a last resort, skip it by resetting the group's offset for that partition by one (Day 11), recording what was skipped.

Check yourself

Lowering min.insync.replicas to 1 on topics produced with acks=all. A write can then be acknowledged with only the leader holding it, so losing that one broker loses acknowledged data, even though the replication factor is 3.

That partition's consumer instance (crashes, a stuck poison message) and whether a hot key sends a disproportionate share of traffic to it.

Kafka processes a partition in order, so one record that always fails blocks every record behind it. Sending it to a dead-letter topic, with the error and original coordinates, lets the partition move on, and the bad record can be inspected and replayed once the bug is fixed.

Replication factor 3 with min.insync.replicas=2 and acks=all, idempotent producers, manual offset commits after processing, a dead-letter strategy, schemas in a registry, ACLs per service, auto topic creation off, and alerts on offline partitions, under-replication and lag in time.

Steps like resetting offsets, replacing a broker or failing over to DR are rarely needed and easy to get wrong under pressure. Rehearsing them in staging finds missing permissions and outdated commands while the stakes are low.