Kafka Idempotent Producer and Transactions
On this page
How duplicates appear in Kafka
- Producer retries. A producer sends a batch; the broker writes it, but the acknowledgment is lost on the network. The producer retries, and the batch is written twice.
- Crash between output and commit. An app reads from
orders, writes enriched events toorders-enriched, then commits its input offsets. If it crashes after writing but before committing, it re-reads the same inputs on restart and writes the outputs again.
Kafka has a fix for each.
The idempotent producer
With enable.idempotence=true (the default since Kafka 3.0), each producer gets a producer id, and every batch carries a sequence number per partition. The broker remembers the last sequence numbers it accepted and discards a batch it has already written. Retries become safe, and records within a partition stay in order even with several requests in flight.
It requires acks=all, and it covers one producer session's retries only. It doesn't help when your application itself sends the same event twice, or after a crash.
Kafka transactions: consume, process, produce atomically
A transactional producer groups writes to several partitions, and the consumer offsets of the records it read, into one atomic unit: all of it becomes visible, or none of it.
# transform.py: read orders, write enriched events, commit offsets, all in one transaction
import json
from confluent_kafka import Consumer, Producer, KafkaException
consumer = Consumer({
"bootstrap.servers": "localhost:9092",
"group.id": "order-enricher",
"enable.auto.commit": False,
"auto.offset.reset": "earliest",
"isolation.level": "read_committed", # ignore aborted upstream transactions
})
producer = Producer({
"bootstrap.servers": "localhost:9092",
"transactional.id": "order-enricher-1", # stable across restarts of this instance
})
producer.init_transactions()
consumer.subscribe(["orders"])
idle = 0
while idle < 5:
msgs = consumer.consume(num_messages=100, timeout=2.0)
if not msgs:
idle += 1
continue
idle = 0
producer.begin_transaction()
try:
for m in msgs:
try:
event = json.loads(m.value())
except (TypeError, ValueError):
continue # skip non-JSON demo records
event["region"] = "eu" # the "transform"
producer.produce("orders-enriched", key=m.key(), value=json.dumps(event))
producer.send_offsets_to_transaction( # offsets commit only if the outputs do
consumer.position(consumer.assignment()), consumer.consumer_group_metadata())
producer.commit_transaction()
except KafkaException:
producer.abort_transaction() # outputs discarded; inputs will be re-read
raise
consumer.close()Run against a local Kafka 4.3 broker, a read_committed consumer on orders-enriched then saw each order event exactly once, with "region": "eu" added. (A production version would also handle retriable versus fatal transaction errors separately and rebuild state after an abort.)
Transaction coordinator, read_committed and fencing
- A transaction coordinator on the brokers tracks each transaction and writes commit or abort markers into the affected partitions.
- Consumers with
isolation.level=read_committedonly see records from committed transactions. With the defaultread_uncommitted, they'd see aborted records too. - The
transactional.idmust be stable for a given logical producer across restarts. When a new instance starts with the same id,init_transactions()fences the old one: if a zombie instance comes back from a long pause, its writes are rejected.
Costs and limits of Kafka transactions
- Each transaction adds coordinator round trips and markers. Batch many records per transaction; one transaction per record is slow.
read_committedconsumers wait for open transactions to finish, so long transactions add end-to-end latency.- Transactions cover Kafka only. A database write or an HTTP call inside the loop isn't part of the transaction (Day 7). Use idempotent handling or the outbox pattern (Day 19) for those.
Kafka Streams wraps all of this behind one setting, processing.guarantee=exactly_once_v2 (Day 15).
Check yourself
It prevents duplicates from the producer's own retries within a session; it doesn't prevent the application sending the same event twice or repeats after a crash.
So the outputs and the record of which inputs were processed commit atomically; after a crash, either both happened or neither did.
A paused or zombie instance resuming and writing after a replacement has started; its writes are rejected because a newer producer with the same id took over.
Extra round trips to the transaction coordinator, transaction markers written to every partition involved, and higher end-to-end latency, because read_committed consumers wait for commits. They also only cover writes within Kafka, not side effects in other systems.
Each producer gets an id and numbers its records with sequence numbers per partition. If a retry resends a batch the broker already wrote, the broker sees the repeated sequence number and discards it instead of appending a duplicate.