The Dual-Write Problem Explained
On this page
Learning Objective
Recognize the dual-write problem in any code that updates a database and publishes a message, explain why no ordering of the two operations is safe, and understand why two-phase commit is not the practical remedy despite appearing to be the obvious one.
Concept Explanation
Almost every event-driven system contains the same three lines of code: update the database, publish an event, return. It is the most natural way to write it, it works in development, it works in staging, it works in production most of the time, and it is unconditionally broken. This is the dual-write problem, and it is the single most common correctness bug in event-driven architecture.
The problem is that two independent systems are being changed and there is no way to make both changes atomically. Whichever order you choose, there is a window in which a crash leaves them inconsistent. Write to the database first and publish second: the commit succeeds, the process is killed, the network drops, or the broker is unavailable, and the event is never published. The database now says the order was placed and no consumer will ever know, so the payment is never taken, the warehouse never ships, and nothing anywhere reports an error. Publish first and write second: the event goes out, the database transaction then fails on a constraint or a deadlock, and consumers are now reacting to an order that does not exist and never will.
The failure mode is asymmetric in a way that makes it especially dangerous. A lost event produces silence: no exception, no alert, no failed request, just a piece of work that never happens, discovered days later by a customer or a reconciliation report. A phantom event produces downstream activity based on a fact that is not true, which in a payment or inventory context means real money or real stock moves for an order that was never created. Neither is caught by tests, because tests do not usually kill the process between two statements.
It is worth being precise about how often this actually bites, because teams frequently dismiss it. The window is small, typically milliseconds, but it is entered on every single write, and it is widened enormously by exactly the conditions under which systems are already struggling: broker unavailability, network partitions, deployments that terminate processes, autoscaling events, out-of-memory kills, and database failovers. A system doing a million writes a day with a one-millisecond window and occasional broker trouble will lose events regularly, and it will lose the most of them during an incident, which is the worst possible time for inconsistency to be introduced.
The obvious remedy, a distributed transaction spanning the database and the broker using two-phase commit, deserves a clear explanation of why it is not used. It requires both resources to support a transaction manager protocol such as XA, which most modern brokers including Kafka do not offer in a generally usable form. It holds locks across a network round trip, which hurts throughput badly. And it introduces the blocking failure mode that makes the protocol notorious: if the coordinator fails after participants have voted to commit, those participants hold locks and cannot proceed until it returns, so the availability of the whole operation is now bounded by the availability of a single coordinator. Event-driven architecture was adopted largely to escape that kind of coupling, so reintroducing it to publish an event is self-defeating. The practical answers are the outbox and change data capture patterns of the next two lessons, which both work by making the two writes into one write.
| Order of operations | Failure point | Resulting state | How it is detected |
|---|---|---|---|
| Commit, then publish | Crash or broker outage after commit | Database has the change; no consumer ever learns of it | Silently, often days later, by a customer or reconciliation |
| Publish, then commit | Transaction fails after publish | Consumers act on a change that never happened | Downstream side effects for a nonexistent entity |
| Publish inside the transaction | Transaction rolls back after publish | Same as above; the broker has no rollback | Same as above |
| Two-phase commit across both | Coordinator fails after the prepare phase | Participants hold locks and block until it recovers | Immediately, as a stall rather than as corruption |
| Outbox or change data capture | Any of the above | One atomic write; the relay retries publication | Not applicable; the inconsistency cannot occur |
Worked Example: The Broken Version, and Why Each Attempted Fix Fails
Each version below is an honest attempt to fix the previous one, and each one still has a window. The problem is not the ordering; it is that there are two writes at all.
// ATTEMPT 1: the version almost everyone writes first.
async function placeOrder(cmd) {
const order = await db.orders.insert(cmd); // committed
await broker.publish('OrderPlaced', order); // <-- crash here = lost forever
return order;
}
// ATTEMPT 2: publish first so nothing is lost. Now events can be phantoms.
async function placeOrder(cmd) {
await broker.publish('OrderPlaced', cmd); // consumers start reacting
const order = await db.orders.insert(cmd); // <-- constraint violation here
return order; // = event about a nonexistent order
}
// ATTEMPT 3: publish inside the transaction. The broker has no rollback, so
// the event is already gone when the transaction is undone.
async function placeOrder(cmd) {
return db.transaction(async (tx) => {
const order = await tx.orders.insert(cmd);
await broker.publish('OrderPlaced', order); // not transactional
await tx.checkCreditLimit(order); // <-- throws, rolls back the insert
return order; // but NOT the publish
});
}
// ATTEMPT 4: retry the publish on failure. Better, but the process can die
// during the retry loop, and the retry state itself lives only in memory.
async function placeOrder(cmd) {
const order = await db.orders.insert(cmd);
await retry(() => broker.publish('OrderPlaced', order), { attempts: 5 });
return order; // <-- SIGKILL mid-retry = lost
}
// The pattern in every failure is identical: durable state exists in one system
// that the other system has no record of. The fix is not a better ordering or a
// better retry. The fix is to make the two writes into a single atomic write,
// which is what the outbox pattern does.Real-World Examples
- Chris Richardson's microservices pattern catalogue documents the dual-write problem and the transactional outbox as its standard remedy, and the pattern's prominence there reflects how frequently it appears in real incident reports: a service that updated its database and published an event, and a percentage of events that quietly went missing during deployments and broker restarts. The catalogue's framing, that reliably updating state and publishing a message is the core problem of event-driven microservices, matches what teams find in practice.
- The X/Open XA specification for distributed transactions, published in 1991, defined the two-phase commit interface that databases and message queues such as IBM MQ implemented for decades. Its decline in modern architectures is instructive: Kafka never offered XA participation in a broadly usable form, cloud-managed services generally do not, and the blocking behavior on coordinator failure was a persistent operational problem, which is why the industry converged on log-based approaches that avoid needing a distributed transaction at all.
Review Questions & Answers
Because both orderings leave a window in which one system has durable state the other does not know about. Committing first risks a lost event, where the change is real but nobody downstream is told, producing silent incompleteness. Publishing first risks a phantom event, where consumers act on a change that was subsequently rolled back and never existed, producing downstream side effects for a nonexistent entity. Which is worse depends on the domain, but neither is acceptable, and there is no third ordering, because the fundamental issue is that two independent durable systems are being changed by two separate operations.
Because a lost event is silent. A phantom event causes downstream activity that eventually hits something that does not exist, a missing order id, a failed lookup, a foreign key violation, so it tends to produce errors that alert somebody. A lost event produces no error anywhere: the write succeeded, the request returned successfully, and the only symptom is work that never happens. Nothing in the system is in an error state, so the discovery path is a customer complaint or a reconciliation report days later, by which time the affected population can be large and the root cause is hard to reconstruct.
Three reasons compounded. It requires both participants to implement a transaction manager protocol, and the brokers that came to dominate, Kafka above all, never offered usable XA participation, so the option often does not exist. It holds locks across network round trips through a prepare and a commit phase, which reduces throughput substantially on the write path. And it blocks: if the coordinator dies between the prepare and the commit, participants are stuck holding locks until it recovers, which makes the availability of every write depend on a single coordinator. Since escaping exactly that kind of coupling is a primary motivation for event-driven architecture, reintroducing it to publish an event defeats the purpose.
They have improved the common case substantially, since most publish failures are transient and a retry succeeds, which reduces the loss rate considerably. What remains is that the retry state lives only in the memory of a process that can be killed at any moment by a deployment, an autoscaler, an out-of-memory condition, or a node failure, and all of those are more likely during the incidents that caused the publish to fail in the first place. The instant the process dies, the pending event is gone with no record that it ever needed publishing. The outbox pattern fixes precisely this by making the intent to publish durable in the same transaction as the state change.