Catalog My learning Blog
Event-Driven Architecture Course: Event Sourcing and CQRS · Phase 4 · Lesson 21 · 8 min read

Change Data Capture (CDC) Explained

On this page
  1. Learning Objective
  2. Concept Explanation
  3. Worked Example: The Same Change as a Raw Row Event and as an Outbox Event
  4. Real-World Examples
  5. Review Questions & Answers

Learning Objective

Explain how log-based change data capture extracts a reliable event stream from a database without application changes, compare it honestly against the outbox pattern, and recognize the leaky-abstraction risk that makes raw table change streams a poor public contract.

Concept Explanation

Every transactional database already maintains an ordered, durable log of every change it has committed, because replication and crash recovery require one: the write-ahead log in PostgreSQL, the binary log in MySQL, the oplog in MongoDB. Change data capture reads that log and turns it into an event stream. The insight is that the database has already solved the hard problem, ordering and durability of committed changes, and the log is sitting there, so an event stream can be derived from it without the application participating at all.

The reliability argument is strong. Because the source is the commit log, an event exists if and only if the transaction committed, which makes lost and phantom events structurally impossible rather than merely unlikely. There is no dual write, because there is only ever one write. It also requires no application change whatsoever, which makes it the only practical option for legacy systems, third-party applications, and databases written to by many services or by hand, and it captures every change including those made by a manual SQL statement at three in the morning during an incident.

The decisive weakness is what the events contain. Raw change data capture emits row-level changes: table, operation, before image, after image. That is a database-internal representation, not a domain event, and publishing it as a public contract couples every consumer to the producer's physical schema. Rename a column, split a table, add a normalization step, and every consumer breaks. It also loses intent: a row where status moved from active to inactive could be a cancellation, an expiry, a suspension, or a data fix, and the row change cannot distinguish them, so downstream consumers guess at business meaning from column values.

This is why the two patterns are best understood as complements rather than alternatives. The strongest common design uses both: the application writes a properly designed domain event to an outbox table inside its transaction, and change data capture tails the log to publish those outbox rows. That combination gets the reliability and low latency of log tailing together with deliberately designed, intent-carrying events, and it is exactly what the Debezium outbox event router implements. Raw table capture without an outbox remains the right choice when the producer cannot be modified, and is best treated as an internal integration mechanism for replication and analytics rather than as a published contract.

Operationally, change data capture brings its own concerns. Initial snapshotting is needed because the log only contains recent changes, so a new consumer must first be seeded with the current table contents and then switched to the live stream at a consistent position. Replication slots must be monitored, since a stalled consumer prevents the database from recycling log segments and can fill the disk of the primary database, which is a serious production hazard specific to this approach. Schema changes flow through and must be handled by consumers. And deletes, transaction boundaries, and large transactions all need explicit consideration, since a single transaction touching a million rows produces a million change events.

DimensionTransactional outboxRaw change data captureOutbox plus change data capture
Application change requiredYes - the service writes outbox rowsNone at allYes - but no relay to build
Event qualityDeliberate domain events with intentRow changes with no business meaningDeliberate domain events
Coupling to physical schemaNone - the event is a designed contractTotal - consumers depend on table structureNone
LatencyPoll interval, typically hundreds of millisecondsNear real time from the commit logNear real time
Works on a legacy or third-party systemNoYes - the main reason to choose itNo
Extra operational burdenRelay process, table cleanupReplication slots, snapshots, slot lag on the primaryConnector operations only

Worked Example: The Same Change as a Raw Row Event and as an Outbox Event

Both records describe one customer cancelling one order. Only one of them tells a consumer what happened, and only one of them survives a schema refactor.

SQL
// --- RAW CHANGE DATA CAPTURE: a row diff, with no business meaning. ---
{
  "source": { "db": "shop", "table": "orders", "lsn": 24771120, "txId": 8891 },
  "op": "u",
  "before": { "id": "9f12", "status": "placed", "total_cents": 17700,
              "updated_at": "2026-03-04T10:22:31Z" },
  "after":  { "id": "9f12", "status": "cancelled", "total_cents": 17700,
              "updated_at": "2026-03-04T11:02:09Z" }
}
// A consumer must infer: was this a customer cancellation, a fraud block, an
// expiry, or an operator correcting a mistake? The row cannot say. And if the
// column is later renamed to order_status, every consumer breaks at once.

// --- OUTBOX EVENT, CAPTURED BY THE SAME MECHANISM: intent is explicit. ---
{
  "eventId": "7c2a...", "eventType": "OrderCancelled", "eventVersion": 1,
  "aggregateId": "9f12", "aggregateType": "order",
  "correlationId": "txn-9f12", "causationId": "cmd-5510",
  "occurredAt": "2026-03-04T11:02:09Z",
  "data": {
    "orderId": "9f12",
    "reason": "customer_request",        // the intent the row diff could not carry
    "cancelledBy": "customer",
    "refundDueCents": 17700
  }
}

// Debezium connector config for the outbox variant: the connector watches only
// the outbox table and routes each row to a topic by aggregate type, keyed by
// aggregate id so per-order ordering is preserved by the broker.
{
  "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
  "table.include.list": "public.outbox",
  "transforms": "outbox",
  "transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter",
  "transforms.outbox.route.by.field": "aggregate_type",
  "transforms.outbox.table.field.event.key": "aggregate_id"
}

// Monitoring that is not optional on PostgreSQL: an inactive or lagging
// replication slot stops WAL segments being recycled and can fill the primary's
// disk, which takes down the write side entirely.
//   SELECT slot_name, active,
//          pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn))
//     FROM pg_replication_slots;
Change data capture reading a database write-ahead log and publishing to a broker, contrasted along two paths: raw table changes coupling consumers to the physical schema, and outbox rows captured by the same mechanism delivering designed domain events.
Change data capture reading a database write-ahead log and publishing to a broker, contrasted along two paths: raw table changes coupling consumers to the physical schema, and outbox rows captured by the same mechanism delivering designed domain events.

Real-World Examples

  • Debezium, created at Red Hat and open-sourced in 2016, reads the PostgreSQL write-ahead log, the MySQL binary log, and equivalents for several other databases, publishing changes to Kafka through Kafka Connect. It has become the de facto standard for change data capture in event-driven systems, and its documentation is unusually candid about the operational concerns this lesson raises, particularly initial snapshots and the danger of an inactive replication slot preventing write-ahead log recycling on a busy primary.
  • Airbnb's SpinalTap, LinkedIn's Databus, and Netflix's DBLog are all internally built change data capture systems that were subsequently described publicly, and the recurring motivation across them is the same: a large number of derived systems, search indexes, caches, analytics stores, and downstream services, needed to react to changes in authoritative databases owned by teams that could not all be asked to publish events themselves. Reading the log was the only approach that scaled organizationally as well as technically.
WarningPublishing raw table change events as a public integration contract binds every consumer to your physical schema, so a column rename or a table split becomes a breaking change for teams you cannot enumerate. Use change data capture as the transport, and an outbox table as the contract, so the events you publish are designed rather than leaked.

Review Questions & Answers

Because there is only one write. The application commits a transaction, and the database's own log records that commit as part of the durability mechanism the database already needs for replication and crash recovery. The event stream is derived from that log, so an event exists exactly when the transaction committed, and there is no second operation that could fail independently. It is not a matter of narrowing a window; the window does not exist, because nothing is being written twice.

Intent, and independence from the physical schema. A row diff shows that status changed from placed to cancelled, but not whether the customer cancelled, fraud blocked the order, it expired, or an operator fixed a mistake, and those four cases usually require entirely different downstream behavior. It also exposes the producer's table structure to every consumer, so renaming a column, splitting a table, or normalizing a field becomes a breaking change for consumers the producer cannot enumerate. A designed domain event carries the reason, the actor, and any derived facts consumers need, and its shape is a contract deliberately chosen to be stable.

Because a replication slot guarantees the database will retain write-ahead log segments until the consumer has read them, which is what makes the stream reliable. If the consumer stops but the slot remains, the database cannot recycle those segments, so write-ahead log storage grows without bound as long as writes continue, and on a busy system this can fill the primary database's disk within hours. At that point the database stops accepting writes, so a failure in a derived pipeline has taken down the authoritative write side. Slot lag must therefore be alerted on in the same tier as primary database disk usage.

When the producing application cannot be modified, which is a real and common situation: a vendor application, a legacy system nobody dares change, a database written to by multiple services or by operational scripts, or a migration where the source must be left untouched while consumers are built against it. It is also reasonable for internal, self-owned pipelines such as replication into an analytics warehouse or a search index, where the consumer is owned by the same team, the coupling to the schema is acknowledged, and a schema change can be coordinated. What it should not be is a published contract offered to teams that cannot be told when a column is renamed.