Catalog My learning Blog
System Design Roadmap: 100-Day Course for Engineers · Phase 7 · Lesson 65 · 4 min read

Idempotency in System Design

On this page
  1. The double charge
  2. What is idempotency?
  3. How idempotency keys work
  4. The side effect outside your database
  5. Idempotent message consumers
  6. Design for idempotency from the start
  7. Check yourself

The double charge

A mobile app sends "charge 129 EUR for order 991". The payment service charges the card, but the response is lost when the phone switches from Wi-Fi to cellular. The app sees a timeout and retries. The service, seeing what looks like a new request, charges again. The customer pays twice. Nothing was buggy in the usual sense; retries are correct behavior. The operation just wasn't idempotent.

What is idempotency?

An operation is idempotent if doing it several times has the same effect as doing it once.

OperationIdempotent?
UPDATE accounts SET status = 'active' WHERE id = 5Yes: setting a value twice leaves the same value
DELETE FROM carts WHERE id = 7Yes: the second delete finds nothing
HTTP PUT /users/5 with a full bodyYes, by design
UPDATE accounts SET balance = balance - 50 WHERE id = 5No: each run subtracts again
INSERT INTO payments ... / HTTP POST /paymentsNo: each run creates a new row

Since timeouts can't tell you whether a request was processed (Day 61), and message brokers deliver at least once (Day 42), every operation that might be retried must be idempotent, either naturally or by design.

How idempotency keys work

The standard fix for operations like payments and order creation: the client generates a unique key for each logical operation, such as a UUID, and sends it with the request and every retry of it. The server records each key with its result:

  • First time the key is seen: do the work and store the result with the key, in the same transaction.
  • Key seen before, finished: return the stored result without doing the work again.
  • Key seen before, still in progress: return 409 or ask the client to retry shortly.
  • Same key with a different request body: reject; it's a client bug.
SQL
CREATE TABLE idempotency_keys (
  key          text PRIMARY KEY,
  request_hash text NOT NULL,
  status       int,
  response     jsonb,
  created_at   timestamptz NOT NULL DEFAULT now()
);

BEGIN;
INSERT INTO idempotency_keys (key, request_hash) VALUES ('7f3c9a2e-order-991', 'sha256:ab12')
  ON CONFLICT (key) DO NOTHING RETURNING key;              -- no row returned: a repeat, so stop here
INSERT INTO payments (order_id, amount) VALUES (991, 129.00) RETURNING id;
UPDATE idempotency_keys SET status = 201, response = '{"payment_id": 1}' WHERE key = '7f3c9a2e-order-991';
COMMIT;

Running the retry against PostgreSQL, the second INSERT ... ON CONFLICT DO NOTHING returned no row, the stored response came back, and the payments table still had exactly one row. Because the key and the payment are written in one transaction, there's no window where one exists without the other. The HTTP convention is an Idempotency-Key header, popularized by Stripe's API. Keep keys for a defined period, for example 24 hours, then expire them.

Two identical requests arriving at the same instant are handled too: the second INSERT waits on the primary key until the first transaction commits, then sees the conflict. Scope keys to the caller, such as (customer_id, key), so one client can't collide with or probe another's keys.

The side effect outside your database

The example writes a local payments row. A real payment also calls a provider, and that call isn't inside your transaction. If your service charges the card, then crashes before committing, a retry finds no stored key and charges again. So pass an idempotency key to the provider as well, derived from yours, such as the same UUID. Stripe and most payment APIs accept one, and they'll return the original charge on a repeat. Idempotency has to hold at every system that does real work, not just the first.

A client retries a timed-out request with the same idempotency key; the server returns the stored result instead of repeating the work.
A client retries a timed-out request with the same idempotency key; the server returns the stored result instead of repeating the work.

Idempotent message consumers

The same idea for queues and events:

  • Record processed message ids in the consumer's database, in the same transaction as the side effect, and skip ids already recorded.
  • Use natural keys: "award points for order 991" with a unique constraint on (order_id, reason) can't happen twice.
  • Make updates absolute rather than relative: "set stock to 41 at version 12" instead of "subtract 1".

Design for idempotency from the start

Treat idempotency as a requirement for every endpoint and consumer that changes state, not something to add after the first incident. It's what makes retries (Day 74), at-least-once delivery and sagas (Day 49) safe.

Check yourself

No; each run subtracts again. Wrap it with an idempotency key stored in the same transaction, or record the transaction id with a unique constraint so a repeat is rejected.

Otherwise a crash between them leaves the work done without the key recorded (so a retry repeats it) or the key recorded without the work (so the retry returns a result that doesn't exist).

Recording processed event ids, or a unique natural key for the effect, in the same transaction as the side effect, and skipping duplicates.

The provider was charged, then the service crashed before committing the key, so the retry charged again. Send an idempotency key to the provider too, so it deduplicates the charge.

GET, PUT, DELETE, HEAD and OPTIONS are defined as idempotent: repeating them has the same effect as doing them once. POST and PATCH aren't guaranteed to be, which is why payment and order creation endpoints accept an idempotency key.