Catalog My learning Blog
Apache Kafka Tutorial: Event Streaming to Production · Phase 3 · Lesson 18 · 3 min read

Event Sourcing and CQRS with Kafka

On this page
  1. Kafka for event sourcing and CQRS
  2. Building a read model with Kafka Streams
  3. Where Kafka alone falls short as an event store
  4. What to plan for with event sourcing on Kafka
  5. Check yourself

Kafka for event sourcing and CQRS

An append-only, ordered, retained log of events per key: that's exactly what event sourcing needs (System Design Roadmap, Day 47). Store OrderCreated, ItemAdded, OrderPaid instead of overwriting a row; current state is the fold of those events. Kafka topics keyed by order id hold each order's history in order, and any number of consumers can build views from them.

CQRS (Command Query Responsibility Segregation) is the natural partner: commands produce events, and read models shaped for specific queries are built by consuming them.

Commands produce events to a topic keyed by entity; projections consume the events to build read models such as a KTable, a search index or a database table.
Commands produce events to a topic keyed by entity; projections consume the events to build read models such as a KTable, a search index or a database table.

Building a read model with Kafka Streams

An order summary per customer, kept current from the event stream:

Code
KTable<String, CustomerSummary> summaries = builder
    .stream("order-events", Consumed.with(Serdes.String(), orderEventSerde))  // key: order id
    .selectKey((orderId, e) -> e.customerId())                                 // re-key by customer
    .groupByKey(Grouped.with(Serdes.String(), orderEventSerde))
    .aggregate(
        CustomerSummary::empty,
        (customerId, event, summary) -> summary.apply(event),                  // fold events into state
        Materialized.<String, CustomerSummary, KeyValueStore<Bytes, byte[]>>as("customer-summaries")
            .withValueSerde(summarySerde));

The customer-summaries store can be queried directly (interactive queries, Day 16) or published to a topic and loaded into a database or search index. Need a new view next month? Write a new projection and replay the events from the beginning. That replay ability is the real payoff.

Where Kafka alone falls short as an event store

NeedKafka out of the boxTypical answer
"Load all events for order 991" quicklyNo per-key lookup; you'd scan a partitionKeep a snapshot or state store per key; or use a dedicated event store
Optimistic concurrency ("append only if the stream is at version 7")Not supportedValidate commands against current state in one place, keyed partitioning, or a database as the write model
Keep events foreverPossible (retention.ms=-1), plus tiered storagePlan storage and compaction carefully
Delete personal dataHard in an immutable logKeep personal data out of events, or encrypt per person and delete keys

A common, practical design: the write model lives in a database (or an event store such as KurrentDB, formerly EventStoreDB), which enforces business rules and concurrency, and publishes events through an outbox (Day 19) to Kafka. Kafka then distributes events and feeds read models. Kafka is the backbone, not necessarily the source of truth for each aggregate.

What to plan for with event sourcing on Kafka

  • Schema evolution forever. Old events must stay readable for as long as you might replay them. Use a schema registry with compatibility rules (Day 8).
  • Eventual consistency. A read model lags the write slightly. After a user acts, show them the result of their command, not a possibly stale read.
  • Replays are heavy. Rebuilding a projection from years of events takes time; run it as a new consumer group alongside the old one, then switch.

Check yourself

Kafka topics are ordered, append-only logs that are retained and can be read by many independent consumer groups, each replaying at its own pace. That's exactly what's needed to fan events out to many read models and rebuild them later.

Handling a command usually means loading one aggregate's events quickly and appending new ones only if nobody else appended first. Kafka has no efficient way to read a single key's events out of a large partition, and no compare-and-append on a stream version, so many systems use a database as the store and Kafka for distribution.

Write the new projection and run it as a new consumer group that starts from the earliest offsets, so it replays every retained event and builds its state. This requires that retention, or compaction, kept the events it needs.

Consumers can receive the same event again after a crash or rebalance, before their offset was committed. Storing the last applied offset or event id alongside the read model, in the same transaction, lets a projection skip events it already applied, so replays don't double-count.

A snapshot stores an aggregate's state as of a given event version. Loading the aggregate then means reading the snapshot plus only the events after it, instead of replaying thousands of events every time a command is handled.