Event Replay, Rebuilds and Temporal Queries
On this page
Learning Objective
Use replay deliberately for the four things it is genuinely good for, run a production projection rebuild without a period of serving wrong answers, and recognize the specific hazards that make naive replay dangerous, above all side effects and non-deterministic derivation.
Concept Explanation
Replay is the payoff for everything the event-sourcing phase has asked you to accept. Because the log retains every change in order and nothing is ever overwritten, the system can re-run history through new code, and that capability supports four genuinely distinct uses. The first is rebuilding a projection after a bug fix or a shape change. The second is retroactive analysis, answering a question nobody thought to ask when the schema was designed, by writing a brand-new projection and running it over the whole history. The third is debugging, reproducing an incident exactly by replaying the precise sequence of events that produced it, which is dramatically more reliable than inferring what happened from logs. The fourth is temporal query, reconstructing the state of an aggregate as it existed at any past moment by folding events up to that point and stopping.
Temporal querying deserves emphasis because it is the capability conventional systems simply cannot offer. What was this customer's credit limit on the day we approved that loan, what did the pricing rules say when this order was placed, what did the risk score look like before the model update: in a state-oriented system these questions are answerable only if somebody anticipated them and built an audit table. In an event-sourced system they are the same fold you already run, with a stopping condition. The distinction between the time an event occurred and the time it was recorded matters here too, which is why mature systems store both, allowing questions of the form what did we believe on Tuesday about what happened on Monday.
The dominant hazard in replay is side effects. A command handler that appends an event and also sends an email is fine in normal operation and catastrophic during replay, because replaying a year of history would send a year of emails. The structural defense is a strict separation: event handlers that update derived state are pure functions of the event and are safe to replay any number of times, while handlers that cause external effects are a different category of subscriber, are never invoked during a rebuild, and are typically driven by a distinct subscription that only ever runs from the live position. Any system that intends to replay must be able to state clearly which subscribers are replay-safe and enforce it, rather than relying on whoever is running the rebuild to remember.
The second hazard is non-determinism in derivation. A projection that calls Date.now, generates a random id, looks up a current exchange rate, or calls an external service produces different results on replay than it did originally, which means the rebuilt projection does not match the one it replaced and nobody can tell which is correct. Everything a projection needs must come from the event itself or from data that is equally immutable, which is one of the strongest practical arguments for putting the occurrence timestamp and the relevant contextual facts into the event payload at write time rather than looking them up at read time.
Operationally, a rebuild of a large projection is a data migration and deserves the same care. The standard approach is a shadow rebuild: build the new version into a separate table or index while the old one continues serving reads, monitor progress against the live position, and swap atomically only once the new version has caught up. This avoids the alternative, truncating in place, which serves empty or partial results for however long the rebuild takes. For very large histories, rebuild time itself becomes the constraint, which is where parallel rebuilds partitioned by aggregate id, or periodic projection snapshots that let a rebuild start from a known good checkpoint rather than from zero, start to matter.
| Use of replay | What it requires | Principal hazard |
|---|---|---|
| Rebuild a projection after a bug fix | Deterministic handlers and a checkpoint reset | Serving partial results if rebuilt in place instead of shadowed |
| Answer a new question retroactively | Events that recorded inputs, not just conclusions | The needed facts were never captured in the payload |
| Reproduce an incident for debugging | A replay environment isolated from production side effects | An external call fired during replay against a live system |
| Temporal query at a past moment | Both occurrence time and recording time stored | Confusing when it happened with when we learned it |
| Migrate to a new read store | A shadow build plus an atomic swap | Rebuild time exceeding the window before the next change |
Worked Example: A Shadow Rebuild and a Temporal Query
The rebuild below never disturbs the projection currently serving reads, and the temporal query is the same fold used for rehydration with a stopping condition applied.
// --- Shadow rebuild: build beside the live projection, then swap ---
async function rebuild(name, handlers) {
const shadow = `${name}_rebuild`;
await db.query(`CREATE TABLE ${shadow} (LIKE ${name} INCLUDING ALL)`);
let position = 0;
for await (const batch of store.readAllBatched({ after: 0, size: 5000 })) {
const tx = await db.begin();
for (const event of batch) {
const handler = handlers[event.type];
if (handler) await handler(tx, event, shadow); // pure: no emails, no HTTP
position = event.globalPosition;
}
await tx.commit();
metrics.gauge('rebuild.lag', await store.headPosition() - position);
}
// Catch up the last few events, then swap inside one transaction so readers
// move from the complete old table to the complete new one with no gap.
const tx = await db.begin();
await tx.query(`ALTER TABLE ${name} RENAME TO ${name}_old`);
await tx.query(`ALTER TABLE ${shadow} RENAME TO ${name}`);
await tx.query(
`UPDATE projection_checkpoints SET position = $1 WHERE projection_name = $2`,
[position, name]);
await tx.commit();
await db.query(`DROP TABLE ${name}_old`);
}
// --- Temporal query: state as of a past instant ---
async function stateAsOf(streamId, instant) {
const events = await store.readStream(streamId);
return events
.filter((e) => e.occurredAt <= instant) // occurrence time, not recording time
.reduce((s, e) => apply(s, e), EMPTY);
}
// "What did order 9f12 look like before the payment was captured?" is the
// same fold, stopped earlier. No audit table was ever required to answer it.Real-World Examples
- Kafka's consumer group offset reset is the operational form of replay that most teams meet first: resetting a consumer group to the earliest offset causes the consumer to reprocess the entire retained history. Confluent's documentation and practically every production runbook warn about the same hazard this lesson describes, that any consumer performing external side effects will perform all of them again, which is why teams separate replay-safe projection consumers from effect-producing consumers at the design level.
- Datomic, the database designed by Rich Hickey, exposes temporal query as a first-class API through its as-of and history functions, letting a caller run an ordinary query against the database as it existed at a past transaction or instant. It is a direct demonstration that temporal query is a natural consequence of retaining facts rather than overwriting them, and that once the history is retained the query surface can expose it without anyone building a bespoke audit mechanism.
Review Questions & Answers
Because replay would produce a different result than the original run, which destroys the property that makes projections trustworthy: that they are a deterministic function of the event log. A handler that stamps the current time records when the projection processed the event rather than when the event happened, so a rebuild three years later stamps every historical row with today's date. A handler that calls an external service records whatever that service says now, which may have changed, and also makes the rebuild dependent on that service's availability and rate limits for the entire history. Everything the handler needs must come from the event or from equally immutable data.
Occurrence time is when the fact became true in the real world; recording time is when the system learned about it and appended the event. They differ whenever information arrives late, an offline mobile client syncing, a batch file from a partner, a correction entered a week after the fact. Both are needed for bitemporal questions of the form what did we believe on Tuesday about what happened on Monday, which is exactly the question an auditor, a financial reporter, or an incident investigator asks: not merely what was true, but what the system could have known at the moment it made a decision.
Because they differ entirely in what readers see during the rebuild. Truncating in place means every query against that projection returns empty or partially populated results for the full duration of the rebuild, which for a large history can be hours, and there is no way to abort and return to the previous state once the truncate has happened. A shadow rebuild leaves the live projection serving correct, slightly stale answers throughout, allows the new version to be inspected and compared before it is used, makes the cutover a single atomic rename, and makes aborting free, since the shadow table is simply dropped.
It means the constraint moves from what was stored to what was recorded in the events. If the events captured the inputs to each decision, a new projection written today can be run over the entire history and produce a complete answer as though the system had always been collecting it: cohort behavior across three years, the effect of a pricing rule that was live for six weeks two years ago, a fraud signal defined after the fraud occurred. The limit is real, though, and it is worth stating plainly: replay cannot recover facts that were never written into any event, which is the strongest practical argument for recording inputs rather than conclusions.