Saga Pattern with Compensating Transactions
On this page
Learning Objective
Design a saga as a sequence of local transactions with compensating actions, write compensations that work in a world where nothing can be truly undone, and reason about the intermediate states a saga necessarily exposes to the rest of the system.
Concept Explanation
A business process routinely spans several services: placing an order involves inventory, payment, shipping, and notification, each with its own database. A single transaction across all of them is not available, for all the reasons the dual-write lesson gave. A saga is the established answer: the process is broken into a sequence of local transactions, each atomic within one service, and each step has a compensating action that semantically undoes it if a later step fails.
The compensating action is the heart of the pattern and the part that is most often misunderstood. It is not a rollback, because the earlier transaction has already committed and its effects are already visible to everyone. A compensation is a new forward transaction that counteracts a previous one in business terms: a captured payment is compensated by issuing a refund, not by pretending the capture never happened; reserved inventory is compensated by releasing the reservation; a dispatched shipment is compensated by a recall request, a return label, and possibly a phone call. The record shows both the action and its compensation, which is the honest representation of what actually occurred.
This means sagas are not atomic and cannot be made so. A saga has intermediate states that are visible to the rest of the system, which is the property that makes them harder to design than transactions. During the window, an order exists with payment taken and inventory not yet reserved, and some other part of the system may observe that. The consequences are real and have names in the literature: lost updates, where a concurrent transaction overwrites a saga's write; dirty reads, where another process reads an intermediate state that will later be compensated away; and fuzzy reads, where a value changes mid-saga. Countermeasures exist, semantic locks that mark a record as pending, commutative updates that can be applied in any order, and reread checks before acting, and choosing among them is part of the design rather than an afterthought.
Some steps cannot be compensated at all, and their placement matters enormously. An email cannot be unsent, a payment to an external party may be irreversible, a physical item once shipped cannot be unshipped cheaply. The standard technique is to order the saga so that all reversible or retriable steps come first and the irreversible step comes last, ideally as a pivot: everything before it is compensatable, and everything after it is retriable until it succeeds. A saga designed around a well-chosen pivot has a much simpler failure story than one with an irreversible step in the middle.
Finally, a compensation is a distributed operation and can fail like any other. A refund can fail because the payment provider is down; a release of inventory can fail because the inventory service is unavailable. Compensations must therefore be retriable and idempotent, and they must eventually succeed or escalate, which means a saga needs durable state recording which steps have completed and which compensations are outstanding, a retry policy with limits, and a path to human intervention when automated compensation exhausts its attempts. A saga framework without a stuck-saga queue and an operator interface is incomplete, because the one thing a saga cannot do is give up quietly.
| Saga step | Local transaction | Compensating action | Compensable? |
|---|---|---|---|
| Create order | Insert an order in the pending state | Mark the order cancelled | Yes - fully |
| Reserve inventory | Decrement available and record a reservation | Release the reservation and restore available stock | Yes - fully |
| Capture payment | Charge the customer via the payment provider | Issue a refund, which is visible on the statement | Semantically, not invisibly |
| Dispatch shipment | Create a shipment and hand it to the carrier | Recall request, return label, restocking process | Expensively and partially |
| Send confirmation email | Deliver a message to the customer | None - a second email apologizing | No - place it after the pivot |
Worked Example: An Order Saga With a Pivot and Compensations
The saga below records durable state at every step, so a crash resumes rather than restarting, and its compensations run in reverse order of the steps that actually completed.
CREATE TABLE saga_instances (
saga_id UUID PRIMARY KEY,
saga_type TEXT NOT NULL,
state TEXT NOT NULL, -- running | compensating | done | stuck
current_step INT NOT NULL DEFAULT 0,
completed JSONB NOT NULL DEFAULT '[]',
context JSONB NOT NULL,
attempts INT NOT NULL DEFAULT 0,
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
const ORDER_SAGA = [
{ name: 'createOrder',
run: (c) => orders.create(c.orderId, c.items, { idempotencyKey: c.sagaId }),
compensate: (c) => orders.cancel(c.orderId, 'saga_failed') },
{ name: 'reserveInventory',
run: (c) => inventory.reserve(c.orderId, c.items, { idempotencyKey: c.sagaId }),
compensate: (c) => inventory.release(c.orderId) },
{ name: 'capturePayment',
run: (c) => payments.capture(c.customerId, c.totalCents, { idempotencyKey: c.sagaId }),
compensate: (c) => payments.refund(c.orderId, c.totalCents, { idempotencyKey: `${c.sagaId}-refund` }) },
// PIVOT: everything before here is compensatable; everything after here is
// retried until it succeeds, because it cannot be meaningfully undone.
{ name: 'dispatchShipment', pivot: true,
run: (c) => shipping.dispatch(c.orderId, c.address, { idempotencyKey: c.sagaId }) },
{ name: 'notifyCustomer',
run: (c) => notifications.orderConfirmed(c.customerId, c.orderId) },
];
async function step(sagaId) {
const saga = await db.sagas.get(sagaId);
const def = ORDER_SAGA[saga.current_step];
try {
await def.run(saga.context);
await db.sagas.advance(sagaId, def.name); // durable: a crash resumes here
} catch (err) {
if (isRetryable(err) || def.pivot) {
// Past the pivot there is no going back, so retry with backoff and
// escalate to a human rather than attempting a compensation.
return db.sagas.scheduleRetry(sagaId, backoff(saga.attempts), { escalateAfter: 8 });
}
await db.sagas.setState(sagaId, 'compensating');
}
}
// Compensations run in reverse order, and ONLY for steps that actually
// completed. Each one is idempotent, because it will certainly be retried.
async function compensate(sagaId) {
const saga = await db.sagas.get(sagaId);
for (const name of [...saga.completed].reverse()) {
const def = ORDER_SAGA.find((s) => s.name === name);
if (!def.compensate) continue;
try {
await def.compensate(saga.context);
await db.sagas.markCompensated(sagaId, name);
} catch (err) {
// A failed compensation is never silently dropped. It is retried, and
// after a bounded number of attempts a human is given the case.
return db.sagas.setState(sagaId, 'stuck', { failedCompensation: name, err });
}
}
await db.sagas.setState(sagaId, 'done');
}Real-World Examples
- The saga concept comes from a 1987 paper by Hector Garcia-Molina and Kenneth Salem at Princeton, written to address long-lived transactions that would otherwise hold database locks for unacceptable periods. Their proposal, a sequence of transactions each with a compensating transaction, was intended for a single database, and the microservices community adopted it decades later for distributed workflows where no shared transaction is possible at all, which is a rather larger jump than the terminology suggests.
- Modern workflow engines including Temporal, AWS Step Functions, Netflix Conductor, and Camunda Zeebe all provide durable execution with built-in compensation support, and their popularity reflects how much of a saga implementation is undifferentiated plumbing: durable state, retries with backoff, timeouts, compensation ordering, and an operator interface for stuck instances. Temporal in particular emphasizes that an unreliable compensation path is the most common defect in hand-rolled sagas, because it is the code path that is least exercised in testing and most needed during incidents.
Review Questions & Answers
A rollback erases an uncommitted change so that no observer ever saw it. A compensation runs after the original transaction has committed and become visible, so it cannot erase anything; it is a new forward transaction that counteracts the earlier one in business terms. The practical differences are substantial: the customer sees both a charge and a refund on their statement rather than nothing at all, other services may already have reacted to the original event, the compensation itself can fail and must be retried, and the audit record honestly shows that the action happened and was then reversed. Designing compensations therefore requires thinking about what the business wants to have happened, not merely how to undo a write.
The pivot is the step that cannot be compensated, placed deliberately at a specific point in the sequence so that the saga has exactly two failure regimes rather than an arbitrary mixture. Before the pivot, every step is compensatable, so any failure triggers a clean reverse compensation chain. At or after the pivot, no compensation exists, so the only valid response is to retry until the step succeeds or to escalate to a human. This turns failure handling from a step-by-step case analysis into two well-defined behaviors, and it makes the design question explicit: what is the one point past which we are committed?
Because a compensation is a distributed call that can time out without the caller learning the outcome, and the saga will retry it, so it may be executed more than once. A non-idempotent refund issued twice returns the customer's money twice, which is a direct financial loss; a non-idempotent inventory release applied twice restores stock that does not exist and leads to overselling. Passing a deterministic idempotency key derived from the saga id makes the second execution a no-op that returns the original result. The failure is particularly insidious because the compensation path is exercised rarely and almost never under the concurrent, failing conditions that trigger the duplicates.
The classic three are lost updates, where a concurrent transaction overwrites a value the saga wrote and will later need to compensate; dirty reads, where another process observes an intermediate state that is subsequently compensated away and acts on data that ends up never having been true; and fuzzy reads, where a value read twice during the saga differs between reads. The countermeasures are semantic locks, marking a record as pending so other processes know it is mid-workflow and can refuse or wait; commutative updates, designing operations such as increments that produce the same result in any order; and rereading a value immediately before acting on it rather than relying on a value read earlier in the flow.