Catalog My learning Blog
Distributed Systems Explained: CAP, Clocks and Replication · Phase 4 · Lesson 23 · 4 min read

Saga Pattern for Long-Running Transactions

On this page
  1. Booking a trip across three services
  2. An orchestrated saga in Python
  3. Saga orchestration vs choreography
  4. What sagas don't give you: isolation and simple undo
  5. Check yourself

Booking a trip across three services

A trip booking needs a flight, a hotel and a card charge, each owned by a different service with its own database. Two-phase commit (Day 22) would lock all three while waiting on the slowest, and the card processor doesn't support prepare anyway. Bookings may also take minutes, when a supplier responds slowly.

A saga (Garcia-Molina and Salem, 1987) breaks the work into local transactions, each committing on its own. For each step, you write a compensating transaction that semantically undoes it: cancel the flight, cancel the hotel, refund the charge. If a later step fails, run the compensations for the completed steps in reverse order.

Local transactions commit one by one; when the last step fails, compensations run in reverse to undo the steps that already committed.
Local transactions commit one by one; when the last step fails, compensations run in reverse to undo the steps that already committed.

An orchestrated saga in Python

Python
# saga.py: an orchestrated saga that compensates completed steps in reverse
def book_flight(trip):   trip["flight"] = "FL-204";  print("  flight booked")
def cancel_flight(trip): trip.pop("flight");         print("  flight cancelled (compensation)")
def book_hotel(trip):    trip["hotel"] = "HT-77";    print("  hotel booked")
def cancel_hotel(trip):  trip.pop("hotel");          print("  hotel cancelled (compensation)")
def charge_card(trip):   raise RuntimeError("card declined")
def refund_card(trip):   print("  refund issued (compensation)")

STEPS = [(book_flight, cancel_flight), (book_hotel, cancel_hotel), (charge_card, refund_card)]

def run_saga(trip):
    done = []
    for action, compensate in STEPS:
        try:
            action(trip)
            done.append(compensate)
        except Exception as e:
            print(f"  step failed: {e}")
            for comp in reversed(done):
                comp(trip)
            return "compensated"
    return "completed"

trip = {}
print(run_saga(trip), trip)
Code
  flight booked
  hotel booked
  step failed: card declined
  hotel cancelled (compensation)
  flight cancelled (compensation)
compensated {}

The failed charge never committed, so only the hotel and flight are undone. In production, the orchestrator's progress lives in a database, so a crashed orchestrator resumes where it stopped. Workflow engines such as Temporal, AWS Step Functions and Camunda provide this durable state for you.

Saga orchestration vs choreography

OrchestrationChoreography
HowA central orchestrator calls each step and decides what's nextEach service reacts to events (FlightBooked triggers the hotel booking)
VisibilityThe whole flow is in one placeThe flow is spread across services
CouplingServices depend on the orchestratorServices depend on each other's events
Best forFlows with many steps or branchesShort flows with two or three steps

Choreographed services need to publish events reliably with their local commit, which is what the transactional outbox does. Without it, a hotel service that commits a booking and then publishes HotelBooked can crash in between, and the saga silently stops. With it, the event is one more row in the same local transaction (tested on PostgreSQL 16):

SQL
BEGIN;
INSERT INTO bookings (id, trip_id, status) VALUES ('HT-77', 'trip-9', 'PENDING');
INSERT INTO outbox (id, topic, payload)
VALUES (gen_random_uuid(), 'hotel-events', '{"type": "HotelBooked", "trip": "trip-9", "booking": "HT-77"}');
COMMIT;

Either both rows exist or neither does. A separate relay reads unsent outbox rows, publishes them to the broker and marks them sent, by polling the table or by reading the database's change log (Debezium does this). A relay that crashes after publishing but before marking will publish that event again, so consumers deduplicate by event id (Day 21). The Kafka course's outbox lesson builds a complete relay.

What sagas don't give you: isolation and simple undo

  • Isolation. Other requests can see intermediate states: a hotel room is booked, then cancelled a second later. Users may see a confirmation that later disappears. Mitigations: mark records as PENDING until the saga finishes, order steps so the ones most likely to fail run first, and put irreversible steps last.
  • Simple undo. Compensation is a business action, not a rollback. A refund isn't the same as never charging, and an email already sent can't be unsent; you send a correction.
  • Free retries. Steps and compensations are retried after failures, so each must be idempotent (Day 21). A compensation must also succeed eventually; if it can't, the saga needs a human queue.

Check yourself

A separate local transaction that semantically undoes a committed step, such as cancelling a hotel booking or issuing a refund. It doesn't erase history: the booking and its cancellation both exist, and others may have seen the booking in between.

Fewer earlier steps will have committed when it fails, so there's less to compensate and fewer intermediate states visible to others.

Failures and timeouts cause retries, so a step or compensation may run more than once, for example when the orchestrator crashes after a step succeeded but before recording it. Running it twice must have the same effect as running it once.

The event is written to an outbox table in the same local transaction as the booking, and a relay publishes it afterwards, so the event exists if and only if the booking does.