Catalog My learning Blog
System Design Roadmap: 100-Day Course for Engineers · Phase 5 · Lesson 49 · 4 min read

Saga Pattern for Distributed Transactions

On this page
  1. When the third step fails
  2. How the saga pattern works
  3. Orchestration-based saga
  4. Choreography-based saga
  5. Making saga steps safe to retry
  6. Saga timeouts: the step that never answers
  7. Sagas lack isolation: semantic locks
  8. Check yourself

When the third step fails

An order goes through three services: inventory reserves stock, payment charges the card, shipping books a courier. The first two succeed. The courier's API is down. Stock is reserved and the customer is charged for an order that won't ship. There's no database transaction to roll back, because three databases were involved. Day 27 introduced the answer: a saga. This lesson builds one.

How the saga pattern works

A saga is a sequence of local transactions, one per service. Each step commits in its own database and has a compensating action: a new transaction that semantically undoes it.

StepActionCompensation
1Reserve inventoryRelease the reservation
2Charge paymentRefund the payment
3Create shipmentCancel the shipment

If step 3 fails, the saga runs compensations for the completed steps in reverse order: refund, then release stock, then mark the order failed and tell the customer.

Each service commits its own local transaction; a failed step triggers compensating actions in reverse order.
Each service commits its own local transaction; a failed step triggers compensating actions in reverse order.

A compensation isn't a rollback. The charge happened, and a refund is a separate, visible event on the customer's statement. Some steps can't be undone at all, like sending an email, so put them last, after every step that can fail.

Orchestration-based saga

A saga orchestrator stores the saga's state and tells each service what to do:

YAML
state: STARTED
  -> send ReserveStock         <- StockReserved       state: STOCK_RESERVED
  -> send ChargePayment        <- PaymentCharged      state: PAID
  -> send CreateShipment       <- ShipmentFailed
  -> send RefundPayment        <- PaymentRefunded     state: COMPENSATING
  -> send ReleaseStock         <- StockReleased       state: FAILED (customer notified)

The state is persisted after each step, so if the orchestrator crashes it resumes where it left off. The whole process is visible in one place. Workflow engines such as Temporal, AWS Step Functions and Azure Durable Functions provide this durability for you.

Choreography-based saga

No orchestrator. Each service listens for events and emits its own: OrderPlaced → inventory emits StockReserved → payment emits PaymentCharged → shipping emits ShipmentFailed → payment hears it and refunds, emitting PaymentRefunded → inventory hears that and releases stock. Less central coupling; harder to see and reason about as steps grow. Day 46 compares the two styles.

Making saga steps safe to retry

Messages are delivered at least once, and orchestrators retry after timeouts. So:

  • Every step and compensation must be idempotent. "Charge payment for order 991" run twice must charge once. Pass a saga or step id as an idempotency key (Day 65).
  • Compensations must succeed eventually. Retry them with backoff; if they keep failing, alert a human. A saga stuck halfway is worse than either outcome.
  • Prefer reservations to final actions. Authorize the card at step 2 and capture it only after shipment is booked. A failed booking then releases an authorization instead of issuing a refund.
  • Handle the in-between state. For a while, stock is reserved and payment pending. Make that visible and sensible: "Processing your order".

Saga timeouts: the step that never answers

The hardest case isn't a step that fails; it's one that doesn't answer. The orchestrator sends ChargePayment and hears nothing for 30 seconds. The charge may have happened or not. Compensating with a refund for a charge that never happened, or retrying one that did, are both wrong.

So every step needs a timeout, and on timeout the orchestrator first asks for the outcome, using the same idempotency key ("what happened to the charge for saga 991?"). Then it continues or compensates. Many payment APIs support exactly this lookup.

Sagas lack isolation: semantic locks

A database transaction hides its intermediate state from everyone else. A saga doesn't. While saga A has reserved the last unit, saga B sees zero stock and turns a customer away, even though A will release it a second later when its payment fails. Or B reads a balance that A is about to compensate.

The usual countermeasures: mark records with a pending state (a semantic lock, such as an order PENDING that other processes treat carefully), prefer updates that work in any order, such as increments, and re-check values before the final step. In .NET, MassTransit's state machine sagas and Azure Durable Functions keep the saga's state and timeouts for you.

Check yourself

The saga runs compensating actions for the completed steps in reverse order: it refunds or voids the payment, releases the reserved inventory, then marks the order as failed and notifies the customer. Each compensation is a new, forward action, not a rollback, and must be safe to retry.

Messages and commands can be delivered or retried more than once. A repeated step must not charge twice or reserve stock twice.

The charge may never have happened, or may still complete after the refund. Query the payment outcome by idempotency key first, then continue or compensate.

Last, after every step that might still fail. A confirmation email can't be unsent, so if it went out before payment and shipping succeeded, a later failure would leave the customer with a false promise. Ordering irreversible steps at the end means they never need compensating.