Saga Pattern for Distributed Transactions
On this page
When the third step fails
An order goes through three services: inventory reserves stock, payment charges the card, shipping books a courier. The first two succeed. The courier's API is down. Stock is reserved and the customer is charged for an order that won't ship. There's no database transaction to roll back, because three databases were involved. Day 27 introduced the answer: a saga. This lesson builds one.
How the saga pattern works
A saga is a sequence of local transactions, one per service. Each step commits in its own database and has a compensating action: a new transaction that semantically undoes it.
| Step | Action | Compensation |
|---|---|---|
| 1 | Reserve inventory | Release the reservation |
| 2 | Charge payment | Refund the payment |
| 3 | Create shipment | Cancel the shipment |
If step 3 fails, the saga runs compensations for the completed steps in reverse order: refund, then release stock, then mark the order failed and tell the customer.
A compensation isn't a rollback. The charge happened, and a refund is a separate, visible event on the customer's statement. Some steps can't be undone at all, like sending an email, so put them last, after every step that can fail.
Orchestration-based saga
A saga orchestrator stores the saga's state and tells each service what to do:
state: STARTED
-> send ReserveStock <- StockReserved state: STOCK_RESERVED
-> send ChargePayment <- PaymentCharged state: PAID
-> send CreateShipment <- ShipmentFailed
-> send RefundPayment <- PaymentRefunded state: COMPENSATING
-> send ReleaseStock <- StockReleased state: FAILED (customer notified)The state is persisted after each step, so if the orchestrator crashes it resumes where it left off. The whole process is visible in one place. Workflow engines such as Temporal, AWS Step Functions and Azure Durable Functions provide this durability for you.
Choreography-based saga
No orchestrator. Each service listens for events and emits its own: OrderPlaced → inventory emits StockReserved → payment emits PaymentCharged → shipping emits ShipmentFailed → payment hears it and refunds, emitting PaymentRefunded → inventory hears that and releases stock. Less central coupling; harder to see and reason about as steps grow. Day 46 compares the two styles.
Making saga steps safe to retry
Messages are delivered at least once, and orchestrators retry after timeouts. So:
- Every step and compensation must be idempotent. "Charge payment for order 991" run twice must charge once. Pass a saga or step id as an idempotency key (Day 65).
- Compensations must succeed eventually. Retry them with backoff; if they keep failing, alert a human. A saga stuck halfway is worse than either outcome.
- Prefer reservations to final actions. Authorize the card at step 2 and capture it only after shipment is booked. A failed booking then releases an authorization instead of issuing a refund.
- Handle the in-between state. For a while, stock is reserved and payment pending. Make that visible and sensible: "Processing your order".
Saga timeouts: the step that never answers
The hardest case isn't a step that fails; it's one that doesn't answer. The orchestrator sends ChargePayment and hears nothing for 30 seconds. The charge may have happened or not. Compensating with a refund for a charge that never happened, or retrying one that did, are both wrong.
So every step needs a timeout, and on timeout the orchestrator first asks for the outcome, using the same idempotency key ("what happened to the charge for saga 991?"). Then it continues or compensates. Many payment APIs support exactly this lookup.
Sagas lack isolation: semantic locks
A database transaction hides its intermediate state from everyone else. A saga doesn't. While saga A has reserved the last unit, saga B sees zero stock and turns a customer away, even though A will release it a second later when its payment fails. Or B reads a balance that A is about to compensate.
The usual countermeasures: mark records with a pending state (a semantic lock, such as an order PENDING that other processes treat carefully), prefer updates that work in any order, such as increments, and re-check values before the final step. In .NET, MassTransit's state machine sagas and Azure Durable Functions keep the saga's state and timeouts for you.
Check yourself
The saga runs compensating actions for the completed steps in reverse order: it refunds or voids the payment, releases the reserved inventory, then marks the order as failed and notifies the customer. Each compensation is a new, forward action, not a rollback, and must be safe to retry.
Messages and commands can be delivered or retried more than once. A repeated step must not charge twice or reserve stock twice.
The charge may never have happened, or may still complete after the refund. Query the payment outcome by idempotency key first, then continue or compensate.
Last, after every step that might still fail. A confirmation email can't be unsent, so if it went out before payment and shipping succeeded, a later failure would leave the customer with a false promise. Ordering irreversible steps at the end means they never need compensating.