Catalog My learning Blog
Apache Kafka Tutorial: Event Streaming to Production · Phase 2 · Lesson 7 · 6 min read

Kafka Delivery Semantics: At-Least vs Exactly-Once

On this page
  1. At-most-once vs at-least-once vs exactly-once
  2. Why enable.auto.commit is risky
  3. At-least-once processing with manual commits
  4. Storing offsets in the database instead
  5. Poison messages and dead-letter topics
  6. Exactly-once semantics in Kafka: what it covers
  7. Choosing a delivery guarantee
  8. Check yourself

At-most-once vs at-least-once vs exactly-once

A consumer reads a payment confirmation, updates an order and commits its offset. If it crashes in the middle, what happens to that record when the consumer restarts? The answer depends on one choice: commit before or after processing.

GuaranteeCommit happensAfter a crashRisk
At most onceBefore processingThe record is skippedRecords can be lost
At least onceAfter processingThe record is processed againRecords can be processed twice
Exactly onceAtomically with the resultNeither lost nor repeatedOnly within Kafka, or with extra work
Committing before processing can lose a record on a crash; committing after can process it twice; exactly-once makes the commit and the result atomic.
Committing before processing can lose a record on a crash; committing after can process it twice; exactly-once makes the commit and the result atomic.

Why enable.auto.commit is risky

By default, enable.auto.commit=true commits offsets in the background every 5 seconds, regardless of whether processing finished. A crash can then lose records (committed but not processed) or repeat them (processed but not yet committed). For anything that matters, turn it off and commit yourself.

At-least-once processing with manual commits

Process first, commit after, and make processing idempotent so a repeat is harmless:

Code
# consumer.py: at-least-once processing with manual commits
import json
from confluent_kafka import Consumer, KafkaException

consumer = Consumer({
    "bootstrap.servers": "localhost:9092",
    "group.id": "payment-confirmations",
    "enable.auto.commit": False,          # commit only after processing succeeds
    "auto.offset.reset": "earliest",      # a brand-new group starts at the beginning
})
consumer.subscribe(["orders"])

processed = set()                          # stands in for a table of processed keys

def handle(event: dict) -> None:
    key = (event["order_id"], event["status"])
    if key in processed:                   # idempotent: a redelivery changes nothing
        return
    processed.add(key)
    print("handled", key)

idle = 0
try:
    while idle < 5:                        # demo only: stop after ~10 s with nothing new
        msg = consumer.poll(2.0)
        if msg is None:
            idle += 1
            continue
        idle = 0
        if msg.error():
            raise KafkaException(msg.error())
        try:
            handle(json.loads(msg.value()))
        except ValueError:
            continue                       # not JSON: in production, send to a dead-letter topic
        consumer.commit(message=msg, asynchronous=False)   # commit after the work is done
finally:
    consumer.close()

Run against the orders topic from Day 5, the first run printed handled (991, 'created'), handled (991, 'paid') and handled (992, 'created'). A second run printed nothing: the group resumed after its committed offsets.

In a real service, the "processed" set is a database table with a unique constraint, written in the same transaction as the business change, so the check survives restarts (the idempotency pattern from the System Design Roadmap, Day 65). Committing after every record is simple but slow; commit every N records or every few seconds, accepting that up to N records may be reprocessed after a crash.

Storing offsets in the database instead

When the result lands in a database, you can make the offset part of the same transaction, so the result and "I processed this" can never disagree. Keep a table of offsets, update it in the transaction that applies the event, and on startup tell the consumer where to resume:

SQL
CREATE TABLE consumer_offsets (
  topic       text   NOT NULL,
  part        int    NOT NULL,
  next_offset bigint NOT NULL,
  PRIMARY KEY (topic, part)
);
SQL
def on_assign(consumer, partitions):
    with db.cursor() as cur:
        for p in partitions:
            cur.execute("SELECT next_offset FROM consumer_offsets WHERE topic = %s AND part = %s",
                        (p.topic, p.partition))
            row = cur.fetchone()
            if row:
                p.offset = row[0]          # resume right after the last applied event
    consumer.assign(partitions)            # partitions without a row fall back to auto.offset.reset

consumer.subscribe(["orders"], on_assign=on_assign)

# for each message, in ONE database transaction:
with db:
    with db.cursor() as cur:
        apply_business_change(cur, json.loads(msg.value()))
        cur.execute("""INSERT INTO consumer_offsets (topic, part, next_offset) VALUES (%s, %s, %s)
                       ON CONFLICT (topic, part) DO UPDATE SET next_offset = EXCLUDED.next_offset""",
                    (msg.topic(), msg.partition(), msg.offset() + 1))

A crash before the commit rolls back both the change and the offset, and the record is read again; a crash after it leaves both in place. That's exactly-once effects for this database, without idempotency keys, at the cost of one more table and doing commits yourself.

Poison messages and dead-letter topics

At-least-once has a failure mode: a record that can never be processed, such as malformed JSON or an order referencing a product that doesn't exist. Retrying it forever blocks its partition, because the consumer can't commit past it. The standard fix is a dead-letter topic (DLT): after a few attempts, write the record somewhere else with the reason, commit, and move on.

Code
# retry briefly for errors that might pass, dead-letter the rest, then commit
class TransientError(Exception):         # raise this from handle() for timeouts and the like
    pass

MAX_ATTEMPTS = 3
dlt = Producer({"bootstrap.servers": "localhost:9092", "acks": "all"})

def process_or_dead_letter(msg) -> None:
    error = None
    for attempt in range(MAX_ATTEMPTS):
        try:
            handle(json.loads(msg.value()))
            return                                   # success
        except TransientError as e:                  # e.g. a database timeout: try again
            error = e
            time.sleep(0.5 * 2 ** attempt)
        except Exception as e:                       # bad data: retrying won't help
            error = e
            break
    dlt.produce("orders.dlt", key=msg.key(), value=msg.value(), headers={
        "dlt.original.topic": msg.topic(),
        "dlt.original.partition": str(msg.partition()),
        "dlt.original.offset": str(msg.offset()),
        "dlt.error": repr(error)[:500],
    })
    if dlt.flush(10) != 0:                           # not stored: don't commit past it
        raise RuntimeError("could not write to the dead-letter topic")

# in the loop: process_or_dead_letter(msg), then consumer.commit(message=msg, asynchronous=False)

Three things to decide up front. Alert on the DLT's message rate, because a dead-lettered record is a business event nobody handled. Give someone a way to replay records from the DLT once the bug is fixed. And know that dead-lettering breaks per-key ordering: if order 991's created event is parked, its paid event is processed next. Where that matters, park later records for the same key too, or stop the partition and alert instead. Keep in-loop retries short, so the total stays well under max.poll.interval.ms (Day 6); for longer delays, use separate retry topics that a consumer reads after a wait.

Exactly-once semantics in Kafka: what it covers

Kafka's exactly-once semantics combine the idempotent producer with transactions (Day 13): a consume-process-produce application writes its output records and its consumer offsets in one atomic transaction. Downstream consumers with isolation.level=read_committed see either all of it or none. Kafka Streams turns this on with one setting (processing.guarantee=exactly_once_v2).

The limit: it covers reading from Kafka and writing to Kafka. When the side effect is outside Kafka (a database update, an email, a payment API call), Kafka can't make it atomic with the offset commit. There you get exactly-once effects by combining at-least-once delivery with idempotent processing, or by storing offsets in the same database transaction as the result.

Choosing a delivery guarantee

  • At most once: metrics or logs where an occasional gap doesn't matter and duplicates would.
  • At least once + idempotent handling: the default for almost everything else.
  • Kafka transactions: Kafka-to-Kafka pipelines and stream processing where duplicates in output topics matter.

Check yourself

Auto-commit commits offsets on a timer, every five seconds by default, regardless of whether processing finished. A crash after a commit but before processing loses those records; a crash before the commit replays them. Commit manually after processing for predictable at-least-once behavior.

Not by themselves; the charge is outside Kafka. Use at-least-once with an idempotency key on the payment so a repeated record doesn't charge again.

The consumer only returns records from transactions that committed, and skips records from aborted transactions or ones still in progress. It's required for exactly-once pipelines, so downstream consumers never act on results that were later rolled back.

After a few attempts, write the record to a dead-letter topic with its original topic, partition, offset and error, commit past it, and alert so someone can fix and replay it.