Exactly-Once Semantics and Idempotency in Practice
On this page
The lost response
A client sends "charge $49.99". The server charges the card, then the response is lost: a timeout, a dropped connection, a load balancer restart. The client can't tell whether the charge happened (Day 1). If it doesn't retry, the customer may not be charged. If it retries, they may be charged twice.
Every distributed system faces this on every call that changes state.
At-most-once, at-least-once and exactly-once
| Guarantee | How | Risk |
|---|---|---|
| At most once | Send once, never retry | Lost messages are lost work |
| At least once | Retry until acknowledged | Duplicates |
| Exactly once (delivery) | Not achievable over an unreliable network | The sender can never know whether a lost acknowledgment means a lost message |
What systems really offer is at-least-once delivery plus idempotent processing, which gives exactly-once effects: the message may arrive twice, but the second arrival changes nothing.
What makes an operation idempotent
An operation is idempotent if doing it twice has the same effect as doing it once. Some operations are naturally idempotent: SET balance = 100, DELETE order 42, PUT /users/7 with a full body. Others aren't: balance = balance + 10, INSERT a charge, POST /payments.
To make a non-idempotent operation safe, give each logical request a unique idempotency key, chosen by the client and reused on every retry. The server remembers the keys it has processed and the response it sent.
Idempotency keys: a safely retried charge
The key detail: record the charge and the stored response in the same transaction, so there's no moment where one exists without the other.
# idem.py: an idempotency-key table that makes a retried charge safe
import sqlite3, json, hashlib
db = sqlite3.connect(":memory:")
db.executescript("""
CREATE TABLE charges (id INTEGER PRIMARY KEY, customer TEXT, amount INTEGER);
CREATE TABLE idempotency_keys (key TEXT PRIMARY KEY, request_hash TEXT, response TEXT);
""")
class KeyReused(Exception): pass
def charge(key, customer, amount):
request_hash = hashlib.sha256(json.dumps([customer, amount]).encode()).hexdigest()
with db: # one transaction: the charge and the stored response commit together
row = db.execute("SELECT request_hash, response FROM idempotency_keys WHERE key = ?", (key,)).fetchone()
if row:
if row[0] != request_hash: # same key, different request: a client bug
raise KeyReused("this key was used for a different request")
return json.loads(row[1]) | {"replayed": True}
cur = db.execute("INSERT INTO charges (customer, amount) VALUES (?, ?)", (customer, amount))
response = {"charge_id": cur.lastrowid, "amount": amount}
db.execute("INSERT INTO idempotency_keys VALUES (?, ?, ?)", (key, request_hash, json.dumps(response)))
return response
print(charge("a1f3-order-981", "cust_7", 4999))
print(charge("a1f3-order-981", "cust_7", 4999)) # the client's retry after a lost response
try:
charge("a1f3-order-981", "cust_7", 9999) # a bug reuses the key for another amount
except KeyReused as e:
print("rejected:", e)
print(db.execute("SELECT COUNT(*) FROM charges").fetchone()[0], "charge(s) recorded"){'charge_id': 1, 'amount': 4999}
{'charge_id': 1, 'amount': 4999, 'replayed': True}
rejected: this key was used for a different request
1 charge(s) recordedThe retry gets the original answer and no second charge. The stored hash of the request catches the other mistake: a client that reuses a key for a different amount gets an error instead of someone else's stored response. The primary key on key also protects against two retries racing: one insert wins, the other fails and can re-read the stored response. Stripe's API works this way with its Idempotency-Key header, keeping keys for at least 24 hours.
Where to apply idempotency
- APIs: accept an idempotency key on every state-changing
POST; return the stored result on repeats. Reject a reused key with a different request body. - Message consumers: store processed message ids (or the source offset) in the same transaction as the side effect. A consumer that writes to a database and commits its offset separately will reprocess after a crash.
- Kafka: idempotent producers stop duplicate writes from producer retries (the broker tracks a producer id and a sequence number per partition and drops a batch it has already written; see Kafka Day 13), and transactions give exactly-once from topic to topic. Effects outside Kafka, such as sending an email, still need idempotency on that side.
- Side effects you can't roll back (emails, third-party calls): pass your key to the third party if it supports one, or record "about to send" and "sent" states so a retry can check.
Limits of idempotency keys
- Keys must be unique per logical operation, generated once by the client, not per attempt.
- The stored-keys table needs an expiry policy that's longer than any retry window.
- Idempotency covers repeats of the same request, not two different requests that conflict; that's a concurrency problem (versions, constraints, Day 18).
Check yourself
If an acknowledgment is lost, the sender can't know whether the message arrived, so it must either risk losing it or risk sending it twice.
Otherwise a crash between them leaves a charge with no record, and the retry charges again, or a record with no charge.
The message is reprocessed and the write repeats; storing the offset or message id with the write, in one transaction, fixes it.
So that a key reused for a different request, usually a client bug, is rejected instead of silently returning the first request's stored response.