Catalog My learning Blog
Event-Driven Architecture Course: Event Sourcing and CQRS · Phase 2 · Lesson 11 · 8 min read

Event Replay, Rebuilds and Temporal Queries

On this page
  1. Learning Objective
  2. Concept Explanation
  3. Worked Example: A Shadow Rebuild and a Temporal Query
  4. Real-World Examples
  5. Review Questions & Answers

Learning Objective

Use replay deliberately for the four things it is genuinely good for, run a production projection rebuild without a period of serving wrong answers, and recognize the specific hazards that make naive replay dangerous, above all side effects and non-deterministic derivation.

Concept Explanation

Replay is the payoff for everything the event-sourcing phase has asked you to accept. Because the log retains every change in order and nothing is ever overwritten, the system can re-run history through new code, and that capability supports four genuinely distinct uses. The first is rebuilding a projection after a bug fix or a shape change. The second is retroactive analysis, answering a question nobody thought to ask when the schema was designed, by writing a brand-new projection and running it over the whole history. The third is debugging, reproducing an incident exactly by replaying the precise sequence of events that produced it, which is dramatically more reliable than inferring what happened from logs. The fourth is temporal query, reconstructing the state of an aggregate as it existed at any past moment by folding events up to that point and stopping.

Temporal querying deserves emphasis because it is the capability conventional systems simply cannot offer. What was this customer's credit limit on the day we approved that loan, what did the pricing rules say when this order was placed, what did the risk score look like before the model update: in a state-oriented system these questions are answerable only if somebody anticipated them and built an audit table. In an event-sourced system they are the same fold you already run, with a stopping condition. The distinction between the time an event occurred and the time it was recorded matters here too, which is why mature systems store both, allowing questions of the form what did we believe on Tuesday about what happened on Monday.

The dominant hazard in replay is side effects. A command handler that appends an event and also sends an email is fine in normal operation and catastrophic during replay, because replaying a year of history would send a year of emails. The structural defense is a strict separation: event handlers that update derived state are pure functions of the event and are safe to replay any number of times, while handlers that cause external effects are a different category of subscriber, are never invoked during a rebuild, and are typically driven by a distinct subscription that only ever runs from the live position. Any system that intends to replay must be able to state clearly which subscribers are replay-safe and enforce it, rather than relying on whoever is running the rebuild to remember.

The second hazard is non-determinism in derivation. A projection that calls Date.now, generates a random id, looks up a current exchange rate, or calls an external service produces different results on replay than it did originally, which means the rebuilt projection does not match the one it replaced and nobody can tell which is correct. Everything a projection needs must come from the event itself or from data that is equally immutable, which is one of the strongest practical arguments for putting the occurrence timestamp and the relevant contextual facts into the event payload at write time rather than looking them up at read time.

Operationally, a rebuild of a large projection is a data migration and deserves the same care. The standard approach is a shadow rebuild: build the new version into a separate table or index while the old one continues serving reads, monitor progress against the live position, and swap atomically only once the new version has caught up. This avoids the alternative, truncating in place, which serves empty or partial results for however long the rebuild takes. For very large histories, rebuild time itself becomes the constraint, which is where parallel rebuilds partitioned by aggregate id, or periodic projection snapshots that let a rebuild start from a known good checkpoint rather than from zero, start to matter.

Use of replayWhat it requiresPrincipal hazard
Rebuild a projection after a bug fixDeterministic handlers and a checkpoint resetServing partial results if rebuilt in place instead of shadowed
Answer a new question retroactivelyEvents that recorded inputs, not just conclusionsThe needed facts were never captured in the payload
Reproduce an incident for debuggingA replay environment isolated from production side effectsAn external call fired during replay against a live system
Temporal query at a past momentBoth occurrence time and recording time storedConfusing when it happened with when we learned it
Migrate to a new read storeA shadow build plus an atomic swapRebuild time exceeding the window before the next change

Worked Example: A Shadow Rebuild and a Temporal Query

The rebuild below never disturbs the projection currently serving reads, and the temporal query is the same fold used for rehydration with a stopping condition applied.

TypeScript
// --- Shadow rebuild: build beside the live projection, then swap ---
async function rebuild(name, handlers) {
  const shadow = `${name}_rebuild`;
  await db.query(`CREATE TABLE ${shadow} (LIKE ${name} INCLUDING ALL)`);

  let position = 0;
  for await (const batch of store.readAllBatched({ after: 0, size: 5000 })) {
    const tx = await db.begin();
    for (const event of batch) {
      const handler = handlers[event.type];
      if (handler) await handler(tx, event, shadow);   // pure: no emails, no HTTP
      position = event.globalPosition;
    }
    await tx.commit();
    metrics.gauge('rebuild.lag', await store.headPosition() - position);
  }

  // Catch up the last few events, then swap inside one transaction so readers
  // move from the complete old table to the complete new one with no gap.
  const tx = await db.begin();
  await tx.query(`ALTER TABLE ${name}  RENAME TO ${name}_old`);
  await tx.query(`ALTER TABLE ${shadow} RENAME TO ${name}`);
  await tx.query(
    `UPDATE projection_checkpoints SET position = $1 WHERE projection_name = $2`,
    [position, name]);
  await tx.commit();
  await db.query(`DROP TABLE ${name}_old`);
}

// --- Temporal query: state as of a past instant ---
async function stateAsOf(streamId, instant) {
  const events = await store.readStream(streamId);
  return events
    .filter((e) => e.occurredAt <= instant)      // occurrence time, not recording time
    .reduce((s, e) => apply(s, e), EMPTY);
}

// "What did order 9f12 look like before the payment was captured?" is the
// same fold, stopped earlier. No audit table was ever required to answer it.
A replay pipeline shown three ways: a shadow projection rebuilt alongside the live one and swapped atomically at catch-up, a temporal fold stopping at a past timestamp to reproduce historical state, and a side-effecting subscriber explicitly excluded from the replay path.
A replay pipeline shown three ways: a shadow projection rebuilt alongside the live one and swapped atomically at catch-up, a temporal fold stopping at a past timestamp to reproduce historical state, and a side-effecting subscriber explicitly excluded from the replay path.

Real-World Examples

  • Kafka's consumer group offset reset is the operational form of replay that most teams meet first: resetting a consumer group to the earliest offset causes the consumer to reprocess the entire retained history. Confluent's documentation and practically every production runbook warn about the same hazard this lesson describes, that any consumer performing external side effects will perform all of them again, which is why teams separate replay-safe projection consumers from effect-producing consumers at the design level.
  • Datomic, the database designed by Rich Hickey, exposes temporal query as a first-class API through its as-of and history functions, letting a caller run an ordinary query against the database as it existed at a past transaction or instant. It is a direct demonstration that temporal query is a natural consequence of retaining facts rather than overwriting them, and that once the history is retained the query surface can expose it without anyone building a bespoke audit mechanism.
WarningBefore a system can safely replay, someone must be able to answer which subscribers are replay-safe and which are not, and the answer must be enforced in code rather than remembered by the person running the rebuild. A single side-effecting handler included in a replay can send a year of emails, re-charge a year of cards, or re-notify a year of webhooks in a few minutes.

Review Questions & Answers

Because replay would produce a different result than the original run, which destroys the property that makes projections trustworthy: that they are a deterministic function of the event log. A handler that stamps the current time records when the projection processed the event rather than when the event happened, so a rebuild three years later stamps every historical row with today's date. A handler that calls an external service records whatever that service says now, which may have changed, and also makes the rebuild dependent on that service's availability and rate limits for the entire history. Everything the handler needs must come from the event or from equally immutable data.

Occurrence time is when the fact became true in the real world; recording time is when the system learned about it and appended the event. They differ whenever information arrives late, an offline mobile client syncing, a batch file from a partner, a correction entered a week after the fact. Both are needed for bitemporal questions of the form what did we believe on Tuesday about what happened on Monday, which is exactly the question an auditor, a financial reporter, or an incident investigator asks: not merely what was true, but what the system could have known at the moment it made a decision.

Because they differ entirely in what readers see during the rebuild. Truncating in place means every query against that projection returns empty or partially populated results for the full duration of the rebuild, which for a large history can be hours, and there is no way to abort and return to the previous state once the truncate has happened. A shadow rebuild leaves the live projection serving correct, slightly stale answers throughout, allows the new version to be inspected and compared before it is used, makes the cutover a single atomic rename, and makes aborting free, since the shadow table is simply dropped.

It means the constraint moves from what was stored to what was recorded in the events. If the events captured the inputs to each decision, a new projection written today can be run over the entire history and produce a complete answer as though the system had always been collecting it: cohort behavior across three years, the effect of a pricing rule that was live for six weeks two years ago, a fraud signal defined after the fraud occurred. The limit is real, though, and it is worth stating plainly: replay cannot recover facts that were never written into any event, which is the strongest practical argument for recording inputs rather than conclusions.