Kafka Delivery Semantics: At-Least vs Exactly-Once
On this page
At-most-once vs at-least-once vs exactly-once
A consumer reads a payment confirmation, updates an order and commits its offset. If it crashes in the middle, what happens to that record when the consumer restarts? The answer depends on one choice: commit before or after processing.
| Guarantee | Commit happens | After a crash | Risk |
|---|---|---|---|
| At most once | Before processing | The record is skipped | Records can be lost |
| At least once | After processing | The record is processed again | Records can be processed twice |
| Exactly once | Atomically with the result | Neither lost nor repeated | Only within Kafka, or with extra work |
Why enable.auto.commit is risky
By default, enable.auto.commit=true commits offsets in the background every 5 seconds, regardless of whether processing finished. A crash can then lose records (committed but not processed) or repeat them (processed but not yet committed). For anything that matters, turn it off and commit yourself.
At-least-once processing with manual commits
Process first, commit after, and make processing idempotent so a repeat is harmless:
# consumer.py: at-least-once processing with manual commits
import json
from confluent_kafka import Consumer, KafkaException
consumer = Consumer({
"bootstrap.servers": "localhost:9092",
"group.id": "payment-confirmations",
"enable.auto.commit": False, # commit only after processing succeeds
"auto.offset.reset": "earliest", # a brand-new group starts at the beginning
})
consumer.subscribe(["orders"])
processed = set() # stands in for a table of processed keys
def handle(event: dict) -> None:
key = (event["order_id"], event["status"])
if key in processed: # idempotent: a redelivery changes nothing
return
processed.add(key)
print("handled", key)
idle = 0
try:
while idle < 5: # demo only: stop after ~10 s with nothing new
msg = consumer.poll(2.0)
if msg is None:
idle += 1
continue
idle = 0
if msg.error():
raise KafkaException(msg.error())
try:
handle(json.loads(msg.value()))
except ValueError:
continue # not JSON: in production, send to a dead-letter topic
consumer.commit(message=msg, asynchronous=False) # commit after the work is done
finally:
consumer.close()Run against the orders topic from Day 5, the first run printed handled (991, 'created'), handled (991, 'paid') and handled (992, 'created'). A second run printed nothing: the group resumed after its committed offsets.
In a real service, the "processed" set is a database table with a unique constraint, written in the same transaction as the business change, so the check survives restarts (the idempotency pattern from the System Design Roadmap, Day 65). Committing after every record is simple but slow; commit every N records or every few seconds, accepting that up to N records may be reprocessed after a crash.
Storing offsets in the database instead
When the result lands in a database, you can make the offset part of the same transaction, so the result and "I processed this" can never disagree. Keep a table of offsets, update it in the transaction that applies the event, and on startup tell the consumer where to resume:
CREATE TABLE consumer_offsets (
topic text NOT NULL,
part int NOT NULL,
next_offset bigint NOT NULL,
PRIMARY KEY (topic, part)
);def on_assign(consumer, partitions):
with db.cursor() as cur:
for p in partitions:
cur.execute("SELECT next_offset FROM consumer_offsets WHERE topic = %s AND part = %s",
(p.topic, p.partition))
row = cur.fetchone()
if row:
p.offset = row[0] # resume right after the last applied event
consumer.assign(partitions) # partitions without a row fall back to auto.offset.reset
consumer.subscribe(["orders"], on_assign=on_assign)
# for each message, in ONE database transaction:
with db:
with db.cursor() as cur:
apply_business_change(cur, json.loads(msg.value()))
cur.execute("""INSERT INTO consumer_offsets (topic, part, next_offset) VALUES (%s, %s, %s)
ON CONFLICT (topic, part) DO UPDATE SET next_offset = EXCLUDED.next_offset""",
(msg.topic(), msg.partition(), msg.offset() + 1))A crash before the commit rolls back both the change and the offset, and the record is read again; a crash after it leaves both in place. That's exactly-once effects for this database, without idempotency keys, at the cost of one more table and doing commits yourself.
Poison messages and dead-letter topics
At-least-once has a failure mode: a record that can never be processed, such as malformed JSON or an order referencing a product that doesn't exist. Retrying it forever blocks its partition, because the consumer can't commit past it. The standard fix is a dead-letter topic (DLT): after a few attempts, write the record somewhere else with the reason, commit, and move on.
# retry briefly for errors that might pass, dead-letter the rest, then commit
class TransientError(Exception): # raise this from handle() for timeouts and the like
pass
MAX_ATTEMPTS = 3
dlt = Producer({"bootstrap.servers": "localhost:9092", "acks": "all"})
def process_or_dead_letter(msg) -> None:
error = None
for attempt in range(MAX_ATTEMPTS):
try:
handle(json.loads(msg.value()))
return # success
except TransientError as e: # e.g. a database timeout: try again
error = e
time.sleep(0.5 * 2 ** attempt)
except Exception as e: # bad data: retrying won't help
error = e
break
dlt.produce("orders.dlt", key=msg.key(), value=msg.value(), headers={
"dlt.original.topic": msg.topic(),
"dlt.original.partition": str(msg.partition()),
"dlt.original.offset": str(msg.offset()),
"dlt.error": repr(error)[:500],
})
if dlt.flush(10) != 0: # not stored: don't commit past it
raise RuntimeError("could not write to the dead-letter topic")
# in the loop: process_or_dead_letter(msg), then consumer.commit(message=msg, asynchronous=False)Three things to decide up front. Alert on the DLT's message rate, because a dead-lettered record is a business event nobody handled. Give someone a way to replay records from the DLT once the bug is fixed. And know that dead-lettering breaks per-key ordering: if order 991's created event is parked, its paid event is processed next. Where that matters, park later records for the same key too, or stop the partition and alert instead. Keep in-loop retries short, so the total stays well under max.poll.interval.ms (Day 6); for longer delays, use separate retry topics that a consumer reads after a wait.
Exactly-once semantics in Kafka: what it covers
Kafka's exactly-once semantics combine the idempotent producer with transactions (Day 13): a consume-process-produce application writes its output records and its consumer offsets in one atomic transaction. Downstream consumers with isolation.level=read_committed see either all of it or none. Kafka Streams turns this on with one setting (processing.guarantee=exactly_once_v2).
The limit: it covers reading from Kafka and writing to Kafka. When the side effect is outside Kafka (a database update, an email, a payment API call), Kafka can't make it atomic with the offset commit. There you get exactly-once effects by combining at-least-once delivery with idempotent processing, or by storing offsets in the same database transaction as the result.
Choosing a delivery guarantee
- At most once: metrics or logs where an occasional gap doesn't matter and duplicates would.
- At least once + idempotent handling: the default for almost everything else.
- Kafka transactions: Kafka-to-Kafka pipelines and stream processing where duplicates in output topics matter.
Check yourself
Auto-commit commits offsets on a timer, every five seconds by default, regardless of whether processing finished. A crash after a commit but before processing loses those records; a crash before the commit replays them. Commit manually after processing for predictable at-least-once behavior.
Not by themselves; the charge is outside Kafka. Use at-least-once with an idempotency key on the payment so a repeated record doesn't charge again.
The consumer only returns records from transactions that committed, and skips records from aborted transactions or ones still in progress. It's required for exactly-once pipelines, so downstream consumers never act on results that were later rolled back.
After a few attempts, write the record to a dead-letter topic with its original topic, partition, offset and error, commit past it, and alert so someone can fix and replay it.