Kafka Log Compaction vs Retention
On this page
Two kinds of Kafka topics
An orders topic is a history of events: every created, paid and shipped. After 7 days, old events can go. A user-profiles topic holds the current profile per user: you don't need every past address, but you never want to lose a user's latest profile, however old it is. These need different cleanup policies.
Kafka retention: retention.ms and retention.bytes
cleanup.policy=delete (the default) removes whole old segments:
| Setting | Meaning | Default |
|---|---|---|
retention.ms | Delete segments older than this | 7 days |
retention.bytes | Delete oldest segments when a partition exceeds this | Unlimited |
Deletion works on whole segments (Day 3), and the active segment is never deleted, so data may live somewhat longer than retention.ms. retention.ms=-1 keeps data forever, which is reasonable for an event history you treat as a source of truth, if you plan the storage (tiered storage, Day 22).
Log compaction: keep the latest record per key
cleanup.policy=compact makes a background log cleaner rewrite old segments, keeping only the newest record for each key. The topic becomes a changelog: replaying it from the beginning gives you the latest state of every key.
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic user-profiles \
--partitions 3 --replication-factor 1 --config cleanup.policy=compactTo delete a key, write a tombstone: the key with a null value. Consumers treat it as "this key is gone", and the cleaner eventually removes both the older values and, after delete.retention.ms (1 day by default), the tombstone itself.
Log compaction in practice
On a local broker I wrote u1 (Berlin), u2 (Kai), u1 again (Lisbon), then a tombstone for u2, with settings that make the cleaner run quickly. After some further writes and about 40 seconds:
Partition:1 Offset:0 u1 {"name":"Sam","city":"Berlin"}
Partition:1 Offset:1 u1 {"name":"Sam","city":"Lisbon"}
Partition:0 Offset:2 u2 nullTwo lessons in one output. In partition 0, the earlier u2 record is gone and only the tombstone remains, with its original offset 2: compaction removes records but never renumbers offsets. In partition 1, both u1 values are still there, because nothing new was written to that partition and the cleaner never touches the active segment. Compaction is eventual. Consumers must always handle seeing older values for a key; take the last one.
Settings that control it: segment.ms and segment.bytes (when the active segment rolls and becomes eligible), min.cleanable.dirty.ratio (how much uncompacted data triggers cleaning, 0.5 by default), min.compaction.lag.ms (minimum age before a record can be compacted).
Retention vs compaction: when to use which
| Use | Policy |
|---|---|
| Events, logs, clickstream | delete with retention |
| Current state per key: profiles, settings, prices | compact |
| Kafka Streams state changelogs, Kafka Connect offsets and configs | compact (set up automatically) |
| Latest state, but nothing older than N days | compact,delete |
Compacted topics need keys; a record without a key can't be compacted and is rejected.
Check yourself
A compacted topic. Compaction keeps at least the newest record for every key, however old it is, while discarding older versions. Time-based retention would eventually delete the address of a user who hasn't changed it in three years.
Produce a tombstone: a record with the user's key and a null value. The log cleaner later removes all earlier records for that key, and after delete.retention.ms removes the tombstone itself, so consumers have time to see the deletion.
No. Records in the active segment, or not yet cleaned, aren't compacted; compaction is eventual, so consumers must keep the latest value they see.
Yes, cleanup.policy=compact,delete keeps only the latest value per key and also deletes anything older than the retention period. It suits data where only recent keys matter.