Catalog My learning Blog
Apache Kafka Tutorial: Event Streaming to Production · Phase 3 · Lesson 19 · 5 min read

Transactional Outbox Pattern with Kafka

On this page
  1. The dual-write problem
  2. The transactional outbox pattern
  3. Relaying outbox rows: Debezium CDC or polling
  4. Outbox delivery is at least once: deduplicate in consumers
  5. Cleaning up the outbox table
  6. Why not Kafka transactions?
  7. Check yourself

The dual-write problem

The order service saves an order, then publishes OrderPlaced:

Code
db.save(order)                        // succeeds
kafka.produce("orders", OrderPlaced)  // fails: broker unreachable, or the process crashes here

The order exists, but no event was sent: no email, no fulfillment, no analytics. Swap the order and it's worse: an event announces an order that the database then fails to save. Retrying doesn't fully help, because the process can die between the two steps. Two systems, no shared transaction: that's the dual-write problem.

The transactional outbox pattern

Write the event into an outbox table in the same database transaction as the business change. Then a separate process relays outbox rows to Kafka.

SQL
CREATE TABLE outbox (
  id             uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  aggregatetype  text NOT NULL,         -- e.g. 'order'
  aggregateid    text NOT NULL,         -- becomes the Kafka key: per-order ordering
  type           text NOT NULL,         -- 'OrderPlaced'
  payload        jsonb NOT NULL,
  created_at     timestamptz NOT NULL DEFAULT now()
);

BEGIN;
INSERT INTO orders (id, customer_id, total, status) VALUES (991, 42, 129.00, 'PLACED');
INSERT INTO outbox (aggregatetype, aggregateid, type, payload)
VALUES ('order', '991', 'OrderPlaced', '{"order_id": 991, "customer_id": 42, "total": "129.00"}');
COMMIT;

Run in PostgreSQL, both rows commit together or neither does. The event exists if and only if the order does.

The service writes the order and an outbox row in one transaction; a relay, usually CDC, publishes outbox rows to Kafka.
The service writes the order and an outbox row in one transaction; a relay, usually CDC, publishes outbox rows to Kafka.

Relaying outbox rows: Debezium CDC or polling

Two ways to get outbox rows into Kafka:

Change data capture (recommended). Debezium (Day 12) reads new outbox rows from the database log and publishes them, with Debezium's outbox event router turning each row into a clean event:

JSON
{
  "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
  "table.include.list": "public.outbox",
  "transforms": "outbox",
  "transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter",
  "transforms.outbox.table.field.event.key": "aggregateid",
  "transforms.outbox.route.by.field": "aggregatetype"
}

Each row becomes a record keyed by aggregateid with the payload as its value, routed by default to a topic per aggregate type (outbox.event.order). Database connection settings are as on Day 12.

Polling publisher. A job selects unsent rows in order, publishes them, and marks them sent. Simpler to start with; adds polling delay and load, and needs care to keep order and avoid double-publishing when several instances poll.

If you start with polling, two extra columns make it safe: a seq bigserial for order and a published_at timestamptz that stays null until the row is sent. Run one poller at a time, using a PostgreSQL advisory lock so every service instance can try but only one wins:

SQL
CREATE INDEX outbox_unpublished ON outbox (seq) WHERE published_at IS NULL;

SELECT pg_try_advisory_lock(4242);          -- true for one session; others skip this round

SELECT seq, aggregateid, type, payload
FROM outbox WHERE published_at IS NULL
ORDER BY seq LIMIT 500;

-- produce each row (key = aggregateid), flush the producer, then:
UPDATE outbox SET published_at = now() WHERE seq = ANY(@published_seqs);

The lock belongs to the database session, so if the poller dies, the lock is released and another instance takes over on its next attempt. Select by published_at IS NULL, not "everything after the highest seq I've sent": a transaction can take seq 101 and commit after 102 has already gone out, and a high-water mark would skip 101 forever. That gap is the main reason CDC is the better relay: it reads rows in commit order from the database log.

Outbox delivery is at least once: deduplicate in consumers

The relay can publish a row and crash before recording that it did, so the event may be sent twice. Every outbox event has a unique id; consumers record processed ids and skip repeats (Day 7). That's the whole guarantee: never lost, possibly duplicated, handled idempotently.

Cleaning up the outbox table

The outbox table grows with every event. With CDC, rows can be deleted right after insert (Debezium reads the insert from the log), or cleaned up by a scheduled job after a short retention. Index created_at if you poll.

Why not Kafka transactions?

Kafka transactions (Day 13) make writes within Kafka atomic. They can't include a PostgreSQL commit. The outbox moves the problem into one database transaction, which the database already handles perfectly.

Check yourself

Writing to a database and publishing to Kafka are separate operations without a shared transaction, so a failure between them leaves one done and the other not.

The event row is inserted in the same database transaction as the order, so both commit or neither does; a relay then publishes committed rows.

Yes. If the relay publishes an outbox row to Kafka and crashes before marking it as sent, it publishes it again after restarting. Give every event a unique id and have consumers record processed ids, in the same transaction as their own changes, so duplicates are skipped.

Delete or archive rows once they've been published and a safety window has passed, with a scheduled job or by partitioning the table by time and dropping old partitions. With Debezium CDC, rows can even be deleted right after insert, since the log already captured them.

A Kafka transaction can't include the database write, so it doesn't solve the dual write between your database and Kafka. The outbox puts both the business change and the event in one database transaction.