Catalog My learning Blog
Apache Kafka Tutorial: Event Streaming to Production · Phase 2 · Lesson 10 · 3 min read

Kafka Log Compaction vs Retention

On this page
  1. Two kinds of Kafka topics
  2. Kafka retention: retention.ms and retention.bytes
  3. Log compaction: keep the latest record per key
  4. Log compaction in practice
  5. Retention vs compaction: when to use which
  6. Check yourself

Two kinds of Kafka topics

An orders topic is a history of events: every created, paid and shipped. After 7 days, old events can go. A user-profiles topic holds the current profile per user: you don't need every past address, but you never want to lose a user's latest profile, however old it is. These need different cleanup policies.

Kafka retention: retention.ms and retention.bytes

cleanup.policy=delete (the default) removes whole old segments:

SettingMeaningDefault
retention.msDelete segments older than this7 days
retention.bytesDelete oldest segments when a partition exceeds thisUnlimited

Deletion works on whole segments (Day 3), and the active segment is never deleted, so data may live somewhat longer than retention.ms. retention.ms=-1 keeps data forever, which is reasonable for an event history you treat as a source of truth, if you plan the storage (tiered storage, Day 22).

Log compaction: keep the latest record per key

cleanup.policy=compact makes a background log cleaner rewrite old segments, keeping only the newest record for each key. The topic becomes a changelog: replaying it from the beginning gives you the latest state of every key.

Shell
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic user-profiles \
  --partitions 3 --replication-factor 1 --config cleanup.policy=compact

To delete a key, write a tombstone: the key with a null value. Consumers treat it as "this key is gone", and the cleaner eventually removes both the older values and, after delete.retention.ms (1 day by default), the tombstone itself.

Retention deletes whole old segments by age or size; compaction keeps only the newest record per key, and a tombstone removes a key.
Retention deletes whole old segments by age or size; compaction keeps only the newest record per key, and a tombstone removes a key.

Log compaction in practice

On a local broker I wrote u1 (Berlin), u2 (Kai), u1 again (Lisbon), then a tombstone for u2, with settings that make the cleaner run quickly. After some further writes and about 40 seconds:

Code
Partition:1  Offset:0  u1  {"name":"Sam","city":"Berlin"}
Partition:1  Offset:1  u1  {"name":"Sam","city":"Lisbon"}
Partition:0  Offset:2  u2  null

Two lessons in one output. In partition 0, the earlier u2 record is gone and only the tombstone remains, with its original offset 2: compaction removes records but never renumbers offsets. In partition 1, both u1 values are still there, because nothing new was written to that partition and the cleaner never touches the active segment. Compaction is eventual. Consumers must always handle seeing older values for a key; take the last one.

Settings that control it: segment.ms and segment.bytes (when the active segment rolls and becomes eligible), min.cleanable.dirty.ratio (how much uncompacted data triggers cleaning, 0.5 by default), min.compaction.lag.ms (minimum age before a record can be compacted).

Retention vs compaction: when to use which

UsePolicy
Events, logs, clickstreamdelete with retention
Current state per key: profiles, settings, pricescompact
Kafka Streams state changelogs, Kafka Connect offsets and configscompact (set up automatically)
Latest state, but nothing older than N dayscompact,delete

Compacted topics need keys; a record without a key can't be compacted and is rejected.

Check yourself

A compacted topic. Compaction keeps at least the newest record for every key, however old it is, while discarding older versions. Time-based retention would eventually delete the address of a user who hasn't changed it in three years.

Produce a tombstone: a record with the user's key and a null value. The log cleaner later removes all earlier records for that key, and after delete.retention.ms removes the tombstone itself, so consumers have time to see the deletion.

No. Records in the active segment, or not yet cleaned, aren't compacted; compaction is eventual, so consumers must keep the latest value they see.

Yes, cleanup.policy=compact,delete keeps only the latest value per key and also deletes anything older than the retention period. It suits data where only recent keys matter.