Catalog My learning Blog
Distributed Systems Explained: CAP, Clocks and Replication · Phase 4 · Lesson 21 · 4 min read

Exactly-Once Semantics and Idempotency in Practice

On this page
  1. The lost response
  2. At-most-once, at-least-once and exactly-once
  3. What makes an operation idempotent
  4. Idempotency keys: a safely retried charge
  5. Where to apply idempotency
  6. Limits of idempotency keys
  7. Check yourself

The lost response

A client sends "charge $49.99". The server charges the card, then the response is lost: a timeout, a dropped connection, a load balancer restart. The client can't tell whether the charge happened (Day 1). If it doesn't retry, the customer may not be charged. If it retries, they may be charged twice.

Every distributed system faces this on every call that changes state.

At-most-once, at-least-once and exactly-once

GuaranteeHowRisk
At most onceSend once, never retryLost messages are lost work
At least onceRetry until acknowledgedDuplicates
Exactly once (delivery)Not achievable over an unreliable networkThe sender can never know whether a lost acknowledgment means a lost message

What systems really offer is at-least-once delivery plus idempotent processing, which gives exactly-once effects: the message may arrive twice, but the second arrival changes nothing.

What makes an operation idempotent

An operation is idempotent if doing it twice has the same effect as doing it once. Some operations are naturally idempotent: SET balance = 100, DELETE order 42, PUT /users/7 with a full body. Others aren't: balance = balance + 10, INSERT a charge, POST /payments.

To make a non-idempotent operation safe, give each logical request a unique idempotency key, chosen by the client and reused on every retry. The server remembers the keys it has processed and the response it sent.

A charge succeeds but its response is lost; the client retries with the same idempotency key, and the server returns the stored result instead of charging again.
A charge succeeds but its response is lost; the client retries with the same idempotency key, and the server returns the stored result instead of charging again.

Idempotency keys: a safely retried charge

The key detail: record the charge and the stored response in the same transaction, so there's no moment where one exists without the other.

Code
# idem.py: an idempotency-key table that makes a retried charge safe
import sqlite3, json, hashlib
db = sqlite3.connect(":memory:")
db.executescript("""
CREATE TABLE charges (id INTEGER PRIMARY KEY, customer TEXT, amount INTEGER);
CREATE TABLE idempotency_keys (key TEXT PRIMARY KEY, request_hash TEXT, response TEXT);
""")

class KeyReused(Exception): pass

def charge(key, customer, amount):
    request_hash = hashlib.sha256(json.dumps([customer, amount]).encode()).hexdigest()
    with db:  # one transaction: the charge and the stored response commit together
        row = db.execute("SELECT request_hash, response FROM idempotency_keys WHERE key = ?", (key,)).fetchone()
        if row:
            if row[0] != request_hash:    # same key, different request: a client bug
                raise KeyReused("this key was used for a different request")
            return json.loads(row[1]) | {"replayed": True}
        cur = db.execute("INSERT INTO charges (customer, amount) VALUES (?, ?)", (customer, amount))
        response = {"charge_id": cur.lastrowid, "amount": amount}
        db.execute("INSERT INTO idempotency_keys VALUES (?, ?, ?)", (key, request_hash, json.dumps(response)))
        return response

print(charge("a1f3-order-981", "cust_7", 4999))
print(charge("a1f3-order-981", "cust_7", 4999))   # the client's retry after a lost response
try:
    charge("a1f3-order-981", "cust_7", 9999)       # a bug reuses the key for another amount
except KeyReused as e:
    print("rejected:", e)
print(db.execute("SELECT COUNT(*) FROM charges").fetchone()[0], "charge(s) recorded")
Code
{'charge_id': 1, 'amount': 4999}
{'charge_id': 1, 'amount': 4999, 'replayed': True}
rejected: this key was used for a different request
1 charge(s) recorded

The retry gets the original answer and no second charge. The stored hash of the request catches the other mistake: a client that reuses a key for a different amount gets an error instead of someone else's stored response. The primary key on key also protects against two retries racing: one insert wins, the other fails and can re-read the stored response. Stripe's API works this way with its Idempotency-Key header, keeping keys for at least 24 hours.

Where to apply idempotency

  • APIs: accept an idempotency key on every state-changing POST; return the stored result on repeats. Reject a reused key with a different request body.
  • Message consumers: store processed message ids (or the source offset) in the same transaction as the side effect. A consumer that writes to a database and commits its offset separately will reprocess after a crash.
  • Kafka: idempotent producers stop duplicate writes from producer retries (the broker tracks a producer id and a sequence number per partition and drops a batch it has already written; see Kafka Day 13), and transactions give exactly-once from topic to topic. Effects outside Kafka, such as sending an email, still need idempotency on that side.
  • Side effects you can't roll back (emails, third-party calls): pass your key to the third party if it supports one, or record "about to send" and "sent" states so a retry can check.

Limits of idempotency keys

  • Keys must be unique per logical operation, generated once by the client, not per attempt.
  • The stored-keys table needs an expiry policy that's longer than any retry window.
  • Idempotency covers repeats of the same request, not two different requests that conflict; that's a concurrency problem (versions, constraints, Day 18).

Check yourself

If an acknowledgment is lost, the sender can't know whether the message arrived, so it must either risk losing it or risk sending it twice.

Otherwise a crash between them leaves a charge with no record, and the retry charges again, or a record with no charge.

The message is reprocessed and the write repeats; storing the offset or message id with the write, in one transaction, fixes it.

So that a key reused for a different request, usually a client bug, is rejected instead of silently returning the first request's stored response.