Catalog My learning Blog
Event-Driven Architecture Course: Event Sourcing and CQRS · Phase 2 · Lesson 8 · 8 min read

Event Versioning and Schema Evolution

On this page
  1. Learning Objective
  2. Concept Explanation
  3. Worked Example: An Upcaster Chain Across Three Event Versions
  4. Real-World Examples
  5. Review Questions & Answers

Learning Objective

Design event schemas that can evolve without breaking consumers or historical replay, distinguish backward from forward compatibility and know which one a given change requires, and apply upcasting and the expand-and-contract technique to changes that a compatibility rule alone cannot absorb.

Concept Explanation

Events written five years ago will still be read by code written today, and code written today will still be reading events written five years from now. That bidirectional constraint is what makes event schema evolution genuinely harder than API versioning, where an old client eventually upgrades or is switched off. In event sourcing nothing is ever switched off, because every historical event still has to be foldable by the current reducer.

The vocabulary needs to be precise. Backward compatibility means new code can read old data, which is what replay and rehydration require. Forward compatibility means old code can read new data, which is what a rolling deploy requires, since during the deploy some instances of a consumer are already on the new version while others are not. Full compatibility means both. Adding an optional field with a default is fully compatible. Removing a field or renaming one is neither: old readers expecting the field break on new data, and new readers expecting the new name break on old data. Making an optional field required is backward incompatible for every historical event that omitted it.

The first line of defense is designing events that do not need to change. Events should express business facts in the language of the domain rather than mirroring a database row, because a row shape changes whenever the storage model does while a business fact does not. They should be self-describing, carrying an explicit type and schema version in the envelope so a reader never has to infer what it is holding. They should avoid embedding data that belongs to another service, because you will then be responsible for it changing underneath you. And they should never carry a computed value that a consumer could derive itself, because the day the formula changes you have events in the log with the old arithmetic baked in.

When a change is genuinely unavoidable, there are three established techniques. Weak schema, using a format that tolerates unknown fields and missing optional ones, absorbs the majority of real-world changes for free. Upcasting handles the rest: a small transformation function reads an old event version and produces the current shape in memory at load time, so the reducer only ever sees the current version while the stored events remain untouched. Upcasters chain, so a version 1 event can be upcast to 2 and then to 3, which keeps each transformation small and independently testable. The third is versioning the type itself, emitting OrderPlacedV2 as a distinct type when the semantics, not merely the shape, have changed, since an upcaster can add a field but cannot invent information that was never recorded.

For a change that cannot be expressed as a pure transformation, the expand-and-contract pattern is the standard sequence: first expand by writing both the old and the new shape simultaneously while readers still accept either, then migrate every consumer to the new shape, then contract by ceasing to write the old one. This never leaves a moment where a deployed reader cannot handle what a deployed writer produces. The last resort, and it should genuinely be a last resort, is a stream rewrite: copying every event into a new stream with transformations applied. It breaks immutability, invalidates every checkpoint and snapshot, and requires every downstream consumer to reset, so it is reserved for situations where the alternative is a permanently unmanageable upcaster chain.

ChangeBackward compatible?Forward compatible?Technique
Add an optional field with a defaultYesYesJust do it - this is the goal state
Add a required fieldNo - old events lack itYesUpcaster supplying a default, or make it optional
Remove a fieldYesNo - old readers still expect itExpand and contract; deprecate before removing
Rename a fieldNoNoWrite both names, migrate readers, then drop the old one
Change a field typeNoNoNew field alongside the old, or a new event type version
Change what the event meansNot applicable - a transformation cannot fix semanticsNot applicableEmit a new event type entirely

Worked Example: An Upcaster Chain Across Three Event Versions

The stored events never change. The upcaster chain runs at load time, so the reducer and every projection only ever deal with the current shape, and each step stays small enough to test in isolation.

JavaScript
// v1: what was written in 2023 - a single amount, currency assumed USD.
{ "type": "PaymentCaptured", "eventVersion": 1,
  "data": { "amount": 177.00, "ref": "ch_8812" } }

// v2: 2024, multi-currency shipped. Currency became explicit, and amounts
// moved to integer minor units to eliminate floating point rounding.
{ "type": "PaymentCaptured", "eventVersion": 2,
  "data": { "amountCents": 17700, "currency": "USD", "ref": "ch_8812" } }

// v3: 2026, the processor became explicit because a second one was added.
{ "type": "PaymentCaptured", "eventVersion": 3,
  "data": { "amountCents": 17700, "currency": "USD",
            "processor": "stripe", "processorRef": "ch_8812" } }

// Each upcaster handles exactly one version step. They chain, so a v1 event
// is transformed to v2 and then to v3 before the reducer ever sees it.
const upcasters = {
  'PaymentCaptured': {
    1: (d) => ({ amountCents: Math.round(d.amount * 100), currency: 'USD', ref: d.ref }),
    2: (d) => ({ amountCents: d.amountCents, currency: d.currency,
                 processor: 'stripe', processorRef: d.ref }),
    // 'stripe' is a safe default here only because it is historically true:
    // no other processor existed when v2 events were written. If it were a
    // guess rather than a fact, this would need a new event type instead.
  },
};

function upcast(event, currentVersion) {
  let { data, eventVersion } = event;
  const chain = upcasters[event.type] ?? {};
  while (eventVersion < currentVersion) {
    const step = chain[eventVersion];
    if (!step) throw new Error(`No upcaster for ${event.type} v${eventVersion}`);
    data = step(data);
    eventVersion += 1;
  }
  return { ...event, data, eventVersion };
}
Three stored versions of the same event type flowing through a chain of upcasters at load time, arriving at the reducer in a single current shape, with the stored events themselves shown as unmodified.
Three stored versions of the same event type flowing through a chain of upcasters at load time, arriving at the reducer in a single current shape, with the stored events themselves shown as unmodified.

Real-World Examples

  • Confluent Schema Registry, shipped with the Confluent Platform since 2015 and widely deployed alongside Apache Kafka, enforces exactly the compatibility modes this lesson describes, BACKWARD, FORWARD, FULL, and their transitive variants, rejecting a producer's schema registration if it would violate the configured mode. Its default of BACKWARD compatibility reflects the practical observation that the most common need is for new consumer code to keep reading the entire history of previously written events.
  • Apache Avro, developed within the Hadoop project and now the most common serialization format for Kafka event payloads, was designed from the start around schema resolution: a reader supplies its own schema alongside the writer's schema stored with the data, and the runtime reconciles them by matching field names and applying defaults for fields the reader expects but the writer never wrote. That mechanism is effectively upcasting built into the serialization layer, which is why Avro is so frequently chosen over JSON for long-lived event logs.
Common PitfallNever store a value in an event that a consumer could compute itself, and never shape an event after a database row. Both mistakes guarantee a versioning problem later: the computed value freezes an old formula into permanent history, and the row-shaped event turns every internal storage refactor into a breaking change for every consumer and every replay.

Review Questions & Answers

Backward compatibility, new code reading old data, is required because rehydration and replay must fold events written years ago through today's reducer, and there is no option to retire the old data. Forward compatibility, old code reading new data, is required because rolling deployments mean that for some period both old and new instances of a consumer are running against the same stream, and the old ones will encounter events produced by the new ones. A REST API can usually deprecate and eventually retire an old client version; an event log has no equivalent of retiring history.

An upcaster is a pure transformation of data that is already present, so it can rename, restructure, split, combine, or supply a value that is historically certain. It cannot invent information that was never recorded. You have hit the limit when producing the new shape would require guessing, for example when a new field records which of three payment processors was used and historical events genuinely do not distinguish them. At that point the honest move is a new event type rather than an upcaster that fabricates a plausible-looking value which will later be mistaken for a recorded fact.

Because it is neither backward nor forward compatible: new readers looking for the new name break on every historical event, and old readers looking for the old name break on every newly written event, so there is no single deploy order that avoids a broken state. It requires the full expand-and-contract sequence, writing both names for a period, migrating every reader to the new one, confirming no reader still uses the old, and only then dropping it, which typically spans multiple releases and requires coordination with every consuming team.

Because the computation is a policy that will change, while the stored value is permanent. If a total includes tax under one year's rules and the rules change, every historical event carries a number computed under the old policy, and a replay cannot recompute it because the inputs may not all have been recorded. Consumers then cannot tell which policy produced which value without correlating against deploy dates. Recording the inputs, the line items, quantities, unit prices, and applicable rate, keeps the derivation in code where it can be versioned, corrected, and reasoned about, rather than frozen in an immutable log.