Catalog My learning Blog
Apache Kafka Tutorial: Event Streaming to Production · Phase 2 · Lesson 13 · 4 min read

Kafka Idempotent Producer and Transactions

On this page
  1. How duplicates appear in Kafka
  2. The idempotent producer
  3. Kafka transactions: consume, process, produce atomically
  4. Transaction coordinator, read_committed and fencing
  5. Costs and limits of Kafka transactions
  6. Check yourself

How duplicates appear in Kafka

  1. Producer retries. A producer sends a batch; the broker writes it, but the acknowledgment is lost on the network. The producer retries, and the batch is written twice.
  2. Crash between output and commit. An app reads from orders, writes enriched events to orders-enriched, then commits its input offsets. If it crashes after writing but before committing, it re-reads the same inputs on restart and writes the outputs again.

Kafka has a fix for each.

The idempotent producer

With enable.idempotence=true (the default since Kafka 3.0), each producer gets a producer id, and every batch carries a sequence number per partition. The broker remembers the last sequence numbers it accepted and discards a batch it has already written. Retries become safe, and records within a partition stay in order even with several requests in flight.

It requires acks=all, and it covers one producer session's retries only. It doesn't help when your application itself sends the same event twice, or after a crash.

Kafka transactions: consume, process, produce atomically

A transactional producer groups writes to several partitions, and the consumer offsets of the records it read, into one atomic unit: all of it becomes visible, or none of it.

Code
# transform.py: read orders, write enriched events, commit offsets, all in one transaction
import json
from confluent_kafka import Consumer, Producer, KafkaException

consumer = Consumer({
    "bootstrap.servers": "localhost:9092",
    "group.id": "order-enricher",
    "enable.auto.commit": False,
    "auto.offset.reset": "earliest",
    "isolation.level": "read_committed",          # ignore aborted upstream transactions
})
producer = Producer({
    "bootstrap.servers": "localhost:9092",
    "transactional.id": "order-enricher-1",       # stable across restarts of this instance
})
producer.init_transactions()
consumer.subscribe(["orders"])

idle = 0
while idle < 5:
    msgs = consumer.consume(num_messages=100, timeout=2.0)
    if not msgs:
        idle += 1
        continue
    idle = 0
    producer.begin_transaction()
    try:
        for m in msgs:
            try:
                event = json.loads(m.value())
            except (TypeError, ValueError):
                continue                          # skip non-JSON demo records
            event["region"] = "eu"                # the "transform"
            producer.produce("orders-enriched", key=m.key(), value=json.dumps(event))
        producer.send_offsets_to_transaction(     # offsets commit only if the outputs do
            consumer.position(consumer.assignment()), consumer.consumer_group_metadata())
        producer.commit_transaction()
    except KafkaException:
        producer.abort_transaction()              # outputs discarded; inputs will be re-read
        raise
consumer.close()

Run against a local Kafka 4.3 broker, a read_committed consumer on orders-enriched then saw each order event exactly once, with "region": "eu" added. (A production version would also handle retriable versus fatal transaction errors separately and rebuild state after an abort.)

A transactional producer writes output records and the input offsets in one transaction; read-committed consumers see all of it or none of it.
A transactional producer writes output records and the input offsets in one transaction; read-committed consumers see all of it or none of it.

Transaction coordinator, read_committed and fencing

  • A transaction coordinator on the brokers tracks each transaction and writes commit or abort markers into the affected partitions.
  • Consumers with isolation.level=read_committed only see records from committed transactions. With the default read_uncommitted, they'd see aborted records too.
  • The transactional.id must be stable for a given logical producer across restarts. When a new instance starts with the same id, init_transactions() fences the old one: if a zombie instance comes back from a long pause, its writes are rejected.

Costs and limits of Kafka transactions

  • Each transaction adds coordinator round trips and markers. Batch many records per transaction; one transaction per record is slow.
  • read_committed consumers wait for open transactions to finish, so long transactions add end-to-end latency.
  • Transactions cover Kafka only. A database write or an HTTP call inside the loop isn't part of the transaction (Day 7). Use idempotent handling or the outbox pattern (Day 19) for those.

Kafka Streams wraps all of this behind one setting, processing.guarantee=exactly_once_v2 (Day 15).

Check yourself

It prevents duplicates from the producer's own retries within a session; it doesn't prevent the application sending the same event twice or repeats after a crash.

So the outputs and the record of which inputs were processed commit atomically; after a crash, either both happened or neither did.

A paused or zombie instance resuming and writing after a replacement has started; its writes are rejected because a newer producer with the same id took over.

Extra round trips to the transaction coordinator, transaction markers written to every partition involved, and higher end-to-end latency, because read_committed consumers wait for commits. They also only cover writes within Kafka, not side effects in other systems.

Each producer gets an id and numbers its records with sequence numbers per partition. If a retry resends a batch the broker already wrote, the broker sees the repeated sequence number and discards it instead of appending a duplicate.