Catalog My learning Blog
Event-Driven Architecture Course: Event Sourcing and CQRS · Phase 2 · Lesson 10 · 8 min read

Event Sourcing Projections Explained

On this page
  1. Learning Objective
  2. Concept Explanation
  3. Worked Example: Two Projections Over One Stream, With Transactional Checkpoints
  4. Real-World Examples
  5. Review Questions & Answers

Learning Objective

Explain why an event stream cannot be queried directly, build projections that transform a stream into purpose-shaped queryable state, and reason correctly about checkpoints, idempotency, and the tradeoff between inline and asynchronous projection updates.

Concept Explanation

An event store is excellent at exactly one query: give me the events for this stream, in order. It is terrible at every other query. Find all orders over 500 dollars placed last week by customers in Germany is not answerable without reading every event in the store and folding each stream individually, which is a full scan of the entire history for a question a single indexed table could answer in milliseconds. This is not a deficiency in any particular implementation; it is inherent in storing changes rather than state.

A projection resolves this. It is a process that subscribes to the event stream, applies each event to some queryable store, and maintains that store as a derived view shaped for a particular set of questions. The projection is to the event log what an index is to a table: a redundant, rebuildable structure whose only purpose is to make certain reads fast. The event store answers the question of how we got here; the projection answers the question of what is true now, in whatever shape the reader needs.

Three properties define a well-built projection. It is derived, meaning it can be deleted and rebuilt from the event log at any time with no data loss, which is what makes it safe to change its shape whenever a new query appears. It is purpose-built, meaning it is denormalized and indexed for specific reads rather than being a general normalized model, and it is entirely normal to have five projections over the same events serving five different screens. And it is checkpointed, meaning it durably records the global position it has processed up to, so a restart resumes rather than reprocessing from zero or skipping ahead.

The checkpoint is where correctness lives, and it has the same shape as the idempotency problem from the broker lesson. If the projection writes its state update and then stores its checkpoint as two separate operations, a crash between them causes the event to be reprocessed on restart. That is harmless when the update is idempotent, an upsert keyed by id, a status set to an absolute value, and corrupting when it is not, an increment, an append to a list, a counter. The robust pattern is to write the state change and the checkpoint in the same transaction when both live in the same database, which makes reprocessing impossible, and to make every write idempotent when they do not.

Finally, projections can be updated inline or asynchronously, and the choice determines the consistency the system offers. An inline projection is updated in the same transaction as the event append, so a read immediately after a write sees the change, but the write path now depends on the projection store's availability and pays its latency, and every new projection slows down writes. An asynchronous projection subscribes to the stream and updates independently, which keeps writes fast and lets projections fail, lag, and be rebuilt without touching the write side, at the cost of a consistency window. Most systems that take this pattern seriously choose asynchronous projections by default and deal with the lag explicitly, which is exactly the subject of the CQRS phase that follows.

AspectEvent streamProjection
PurposeRecord what happened, authoritatively and permanentlyAnswer a specific question quickly
ShapeAppend-only, ordered, keyed by streamDenormalized and indexed for particular reads
MutabilityNever modified or deletedFreely rebuilt, reshaped, or dropped
CountExactly one per aggregateAs many as there are distinct query needs
If it is lostCatastrophic - the truth is goneAn inconvenience - replay the log and rebuild
ConsistencyStrongly consistent within an aggregateEventually consistent when updated asynchronously

Worked Example: Two Projections Over One Stream, With Transactional Checkpoints

The same order events feed a per-customer summary and a daily revenue rollup. Neither could be answered from the event store directly, both are disposable, and each keeps its own independent checkpoint so one can be rebuilt without disturbing the other.

JavaScript
CREATE TABLE projection_checkpoints (
  projection_name TEXT PRIMARY KEY,
  position        BIGINT NOT NULL,
  updated_at      TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE customer_order_summary (        -- projection 1: per-customer screen
  customer_id   TEXT PRIMARY KEY,
  order_count   INT    NOT NULL DEFAULT 0,
  lifetime_cents BIGINT NOT NULL DEFAULT 0,
  last_order_at TIMESTAMPTZ
);

CREATE TABLE daily_revenue (                 -- projection 2: finance dashboard
  day          DATE PRIMARY KEY,
  order_count  INT    NOT NULL DEFAULT 0,
  gross_cents  BIGINT NOT NULL DEFAULT 0
);

async function runProjection(name, handlers) {
  const { position } = await db.checkpoints.get(name) ?? { position: 0 };

  for await (const event of store.subscribeAll({ after: position })) {
    const handler = handlers[event.type];
    const tx = await db.begin();
    try {
      if (handler) await handler(tx, event);
      // The state change and the checkpoint commit together. A crash anywhere
      // rolls both back, so the event is reprocessed from a clean position
      // and no increment is ever applied twice.
      await tx.query(
        `INSERT INTO projection_checkpoints (projection_name, position)
         VALUES ($1, $2)
         ON CONFLICT (projection_name) DO UPDATE SET position = $2, updated_at = now()`,
        [name, event.globalPosition],
      );
      await tx.commit();
    } catch (err) {
      await tx.rollback();
      throw err;                 // let supervision restart from the last checkpoint
    }
  }
}

runProjection('customer_order_summary', {
  OrderPlaced: (tx, e) => tx.query(
    `INSERT INTO customer_order_summary (customer_id, order_count, lifetime_cents, last_order_at)
     VALUES ($1, 1, $2, $3)
     ON CONFLICT (customer_id) DO UPDATE
       SET order_count    = customer_order_summary.order_count + 1,
           lifetime_cents = customer_order_summary.lifetime_cents + $2,
           last_order_at  = $3`,
    [e.data.customerId, e.data.totalCents, e.occurredAt]),
});

// Rebuilding either projection is a truncate plus a checkpoint reset to 0.
// Nothing needs to be re-emitted, and the write side never notices.
One ordered event stream feeding three independent projections, each with its own checkpoint position, each writing to a differently shaped queryable store, and one of them shown being truncated and rebuilt from position zero while the others continue.
One ordered event stream feeding three independent projections, each with its own checkpoint position, each writing to a differently shaped queryable store, and one of them shown being truncated and rebuilt from position zero while the others continue.

Real-World Examples

  • Materialize and ksqlDB both productized the projection idea as continuously maintained materialized views over event streams, letting teams declare a SQL view over a Kafka topic that is kept incrementally up to date rather than recomputed on query. Their existence reflects how universal the pattern became: enough teams were hand-writing projection processes with checkpoint tables that maintaining the derived view became a product category of its own.
  • PostgreSQL materialized views are the same concept inside a single database, and their limitations illustrate why event-sourced projections are usually written by hand. A materialized view must be refreshed as a whole or through the more restrictive incremental mechanisms available, whereas a projection over an event stream is updated event by event, which is what makes it possible to keep a large denormalized read model current without periodically recomputing it from scratch.
Best PracticeBuild one projection per question, not one canonical read model that tries to answer everything. Projections are cheap because they are disposable, and a denormalized table shaped exactly for one screen will outperform and outlive a general-purpose model that five screens each use awkwardly.

Review Questions & Answers

Because it removes the possibility of reprocessing rather than tolerating it. With a shared transaction, a crash at any point rolls back both the state change and the checkpoint advance, so the projection restarts from a position at which the event has definitively not been applied, and non-idempotent operations such as increments are perfectly safe. Idempotency is the fallback for when the projection store and the checkpoint store are different systems and cannot share a transaction, and it requires every single handler to be written carefully, whereas a shared transaction makes correctness structural.

It changes how freely a team can change its read models. Because a projection can be dropped and rebuilt from the log, adding an index, denormalizing a new field, splitting one table into two shaped for different screens, or fixing a bug in derivation logic are all routine operations rather than migrations, since the corrected version is simply rebuilt from the events. It also means a new question that nobody anticipated can be answered retroactively over the entire history, which a conventional system could only do if it happened to have retained the necessary data.

Because it puts the projection store back into the write path. The command can no longer succeed unless the projection store is available and fast, so its availability multiplies with the write side's, its latency adds to every write, and each additional projection makes writes slower and more fragile. The whole appeal of asynchronous projections is that the write side commits an event and is done, while the read side catches up on its own schedule and can fail, lag, or be rebuilt without the write path ever knowing.

Fix the handler, truncate the projection tables, reset the checkpoint to zero, and let it rebuild from the event log. It is simple because the events were never wrong, only the derivation was, and the events are still all there in their original form. There is no data to reconstruct, no backup to restore, and no need to ask any producer to re-emit anything. The only real considerations are operational: how long the rebuild takes, and whether to build into a shadow table and swap atomically so readers are not served an empty or partially built projection during the rebuild.