Catalog My learning Blog
Event-Driven Architecture Course: Event Sourcing and CQRS · Phase 4 · Lesson 20 · 8 min read

Transactional Outbox Pattern Explained

On this page
  1. Learning Objective
  2. Concept Explanation
  3. Worked Example: An Outbox Table, a Relay, and the Ordering Guarantee
  4. Real-World Examples
  5. Review Questions & Answers

Learning Objective

Implement the transactional outbox correctly, including the relay process and its ordering and concurrency concerns, explain why it converts an unsolvable atomicity problem into a solvable delivery problem, and account for the at-least-once semantics it necessarily produces.

Concept Explanation

The transactional outbox solves the dual-write problem with one observation: the two writes only conflict because they go to two systems, so write both to one system. The event is inserted into an ordinary table in the same database, in the same transaction, as the business change. Either both are committed or neither is, which is guaranteed by the database rather than by application code, and the atomicity problem disappears entirely.

What remains is a delivery problem, and delivery problems are tractable. A separate relay process reads unpublished rows from the outbox table, publishes each to the broker, and marks it published. If the relay crashes before marking a row, it republishes on restart, producing a duplicate rather than a loss, which is exactly the at-least-once semantics that every event-driven consumer already had to be designed for. The critical asymmetry is that the outbox converts a silent, undetectable loss into a visible duplicate that existing idempotency mechanisms already absorb.

The relay can be built in two styles. Polling is the simple one: query for unpublished rows on an interval, publish, mark them, repeat. It works on any database, is easy to reason about, and its cost is a small constant load plus latency equal to roughly half the poll interval. Log tailing is the more efficient one: the relay reads the database's replication log and sees outbox inserts as they commit, which gives lower latency and no polling load, and is what Debezium does. Polling is the right starting point for most systems, and a poll interval measured in hundreds of milliseconds is usually fine, since the outbox is not meant to carry synchronous work.

Two implementation details cause most of the trouble. The first is ordering: if several relay instances run concurrently, or if a single relay publishes rows in parallel, events for the same aggregate can reach the broker out of order, which breaks consumers that depend on sequence. The standard solutions are to run a single active relay with leader election, to publish in id order and lock rows with a clause such as FOR UPDATE SKIP LOCKED partitioned by aggregate id, or to rely on the broker's per-key ordering by always publishing with the aggregate id as the partition key. The second is table growth: a high-volume outbox accumulates rows quickly, so published rows must be deleted or partitioned away on a schedule, and the query for unpublished rows must be indexed so it never degrades into a scan of published history.

Two variants are worth knowing. The listen-to-yourself pattern skips the separate outbox table by having the service publish the event and consume its own event to perform the database write, which makes the broker the single source of ordering but means the write is not durable until the consumer processes it. And in event-sourced systems the outbox is frequently unnecessary, because the event store is already the durable record of the event, so the relay subscribes directly to the event store rather than to a separate table. Both variants reflect the same underlying principle: there must be exactly one durable write, and everything else must be derived from it.

Relay concernNaive approachWhat goes wrongCorrect approach
OrderingMultiple relay instances publishing in parallelEvents for one aggregate arrive out of orderSingle active relay, or lock per aggregate, or key by aggregate id
Marking publishedMark first, then publishA crash between them loses the event, reintroducing the original bugPublish first, then mark; accept duplicates
Table growthNever deleting published rowsThe unpublished query slows and storage grows without boundScheduled deletion or time-based partitioning
Query costScanning for rows where published is falseFull scans once the table is largeA partial index on unpublished rows only
Relay failureAssuming the relay is always runningEvents accumulate silently with no alertAlert on outbox age and unpublished row count

Worked Example: An Outbox Table, a Relay, and the Ordering Guarantee

The business write and the event insert share one transaction, so neither can exist without the other. Everything after that point is a delivery concern that retries can solve.

JavaScript
CREATE TABLE outbox (
  id            BIGSERIAL PRIMARY KEY,   -- publication order
  aggregate_id  TEXT        NOT NULL,     -- becomes the broker partition key
  aggregate_type TEXT       NOT NULL,
  event_type    TEXT        NOT NULL,
  payload       JSONB       NOT NULL,
  headers       JSONB       NOT NULL,     -- correlationId, causationId
  created_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
  published_at  TIMESTAMPTZ
);

-- Partial index: only unpublished rows are indexed, so the relay's query stays
-- fast no matter how much published history has not yet been cleaned up.
CREATE INDEX outbox_unpublished ON outbox (id) WHERE published_at IS NULL;

-- ONE transaction. The order and the event commit together or not at all.
async function placeOrder(cmd) {
  return db.transaction(async (tx) => {
    const order = await tx.one(
      `INSERT INTO orders (id, customer_id, total_cents, status)
       VALUES ($1, $2, $3, 'placed') RETURNING *`,
      [cmd.orderId, cmd.customerId, cmd.totalCents]);

    await tx.query(
      `INSERT INTO outbox (aggregate_id, aggregate_type, event_type, payload, headers)
       VALUES ($1, 'order', 'OrderPlaced', $2, $3)`,
      [order.id, { orderId: order.id, customerId: order.customer_id,
                   totalCents: order.total_cents },
       { correlationId: cmd.correlationId, causationId: cmd.commandId }]);

    return order;          // no publish here, and no way to lose the event
  });
}

// The relay. SKIP LOCKED lets several instances run without ever handing the
// same row to two of them, and ordering by id preserves publication order.
async function relayOnce() {
  return db.transaction(async (tx) => {
    const rows = await tx.many(
      `SELECT * FROM outbox WHERE published_at IS NULL
        ORDER BY id LIMIT 100 FOR UPDATE SKIP LOCKED`);

    for (const row of rows) {
      // Publish FIRST. A crash here republishes later: a duplicate, not a loss.
      await broker.publish(row.event_type, row.payload, {
        key: row.aggregate_id,        // per-aggregate ordering, enforced by the broker
        headers: row.headers,
      });
      await tx.query('UPDATE outbox SET published_at = now() WHERE id = $1', [row.id]);
    }
  });
}

// Housekeeping and alerting, both mandatory rather than optional:
//   DELETE FROM outbox WHERE published_at < now() - INTERVAL '7 days';
//   alert if max(now() - created_at) for unpublished rows exceeds 60 seconds.
The transactional outbox: one database transaction writing both the business row and the outbox row, a relay process reading unpublished rows in id order and publishing them keyed by aggregate id, and the crash point between publish and mark producing a duplicate that an idempotent consumer absorbs.
The transactional outbox: one database transaction writing both the business row and the outbox row, a relay process reading unpublished rows in id order and publishing them keyed by aggregate id, and the crash point between publish and mark producing a duplicate that an idempotent consumer absorbs.

Real-World Examples

  • Debezium ships an outbox event router as a standard Kafka Connect single message transform, designed specifically for this pattern: the application writes to an outbox table, Debezium captures the inserts from the database transaction log, and the transform routes each row to a topic derived from its aggregate type with the aggregate id as the message key. Its inclusion as a first-class feature reflects how thoroughly the outbox became the default answer to reliable event publication from a transactional service.
  • Chris Richardson's Eventuate framework and the accompanying microservices patterns work popularized the transactional outbox alongside the saga pattern, and the two are almost always deployed together, because a saga's reliability depends entirely on its participants publishing events reliably. A saga built over naive dual writes will strand workflows midway with no record of what happened, which is the failure mode that makes the outbox a prerequisite rather than an enhancement.
Common PitfallIn the relay, always publish before marking the row as published. Marking first turns a crash into a permanently lost event, which is the exact bug the outbox exists to eliminate; publishing first turns the same crash into a duplicate, which every idempotent consumer already handles. Getting this one ordering backwards silently undoes the whole pattern.

Review Questions & Answers

The original problem is that two durable systems must be changed atomically, which is not solvable without a distributed transaction protocol that is unavailable, slow, and blocking. The outbox removes the second system from the critical write by putting the event in the same database, so the database's own transaction guarantees atomicity for free. What is left is getting a durably recorded event from the database to the broker, which is a delivery problem, and delivery problems are solved by retries because the intent is now persistent: the relay can fail any number of times, restart, and still find the row waiting to be published.

Because publishing to the broker and marking the row as published are themselves two writes to two systems, which is the same unsolvable atomicity problem one level down. Choosing to publish first means a crash between the two operations causes the row to be published again on the next pass. That is acceptable because the resulting duplicate is visible and mechanically handled: consumers already need idempotency to cope with broker redelivery and rebalances, so a duplicate from the relay costs nothing extra, whereas the alternative failure, a lost event, is silent and has no general remedy.

Per-aggregate ordering breaks. Two relays can pick up OrderPlaced and OrderCancelled for the same order and publish them in either real order, so a consumer can observe a cancellation for an order it has not yet seen created, which leaves its state machine in an impossible position. The fixes, in ascending order of sophistication, are to run a single active relay chosen by leader election, to lock rows with FOR UPDATE SKIP LOCKED and partition the work by aggregate id so all rows for one aggregate go to one instance, or to rely on the broker's per-key ordering by always publishing with the aggregate id as the partition key so the broker itself serializes them.

Because the count is a function of traffic and says nothing about health: ten thousand rows during a burst may be entirely normal if the relay is clearing them in seconds, while five rows may indicate a total failure if they have been sitting for an hour. Age directly measures the symptom that matters, which is how long an event has been waiting for publication, and it works identically at any traffic level, including at night when a completely stopped relay would show a small and stable row count. The count is still worth graphing for capacity, but the age of the oldest unpublished row is the alert.