Event Sourcing and CQRS with Kafka
On this page
Kafka for event sourcing and CQRS
An append-only, ordered, retained log of events per key: that's exactly what event sourcing needs (System Design Roadmap, Day 47). Store OrderCreated, ItemAdded, OrderPaid instead of overwriting a row; current state is the fold of those events. Kafka topics keyed by order id hold each order's history in order, and any number of consumers can build views from them.
CQRS (Command Query Responsibility Segregation) is the natural partner: commands produce events, and read models shaped for specific queries are built by consuming them.
Building a read model with Kafka Streams
An order summary per customer, kept current from the event stream:
KTable<String, CustomerSummary> summaries = builder
.stream("order-events", Consumed.with(Serdes.String(), orderEventSerde)) // key: order id
.selectKey((orderId, e) -> e.customerId()) // re-key by customer
.groupByKey(Grouped.with(Serdes.String(), orderEventSerde))
.aggregate(
CustomerSummary::empty,
(customerId, event, summary) -> summary.apply(event), // fold events into state
Materialized.<String, CustomerSummary, KeyValueStore<Bytes, byte[]>>as("customer-summaries")
.withValueSerde(summarySerde));The customer-summaries store can be queried directly (interactive queries, Day 16) or published to a topic and loaded into a database or search index. Need a new view next month? Write a new projection and replay the events from the beginning. That replay ability is the real payoff.
Where Kafka alone falls short as an event store
| Need | Kafka out of the box | Typical answer |
|---|---|---|
| "Load all events for order 991" quickly | No per-key lookup; you'd scan a partition | Keep a snapshot or state store per key; or use a dedicated event store |
| Optimistic concurrency ("append only if the stream is at version 7") | Not supported | Validate commands against current state in one place, keyed partitioning, or a database as the write model |
| Keep events forever | Possible (retention.ms=-1), plus tiered storage | Plan storage and compaction carefully |
| Delete personal data | Hard in an immutable log | Keep personal data out of events, or encrypt per person and delete keys |
A common, practical design: the write model lives in a database (or an event store such as KurrentDB, formerly EventStoreDB), which enforces business rules and concurrency, and publishes events through an outbox (Day 19) to Kafka. Kafka then distributes events and feeds read models. Kafka is the backbone, not necessarily the source of truth for each aggregate.
What to plan for with event sourcing on Kafka
- Schema evolution forever. Old events must stay readable for as long as you might replay them. Use a schema registry with compatibility rules (Day 8).
- Eventual consistency. A read model lags the write slightly. After a user acts, show them the result of their command, not a possibly stale read.
- Replays are heavy. Rebuilding a projection from years of events takes time; run it as a new consumer group alongside the old one, then switch.
Check yourself
Kafka topics are ordered, append-only logs that are retained and can be read by many independent consumer groups, each replaying at its own pace. That's exactly what's needed to fan events out to many read models and rebuild them later.
Handling a command usually means loading one aggregate's events quickly and appending new ones only if nobody else appended first. Kafka has no efficient way to read a single key's events out of a large partition, and no compare-and-append on a stream version, so many systems use a database as the store and Kafka for distribution.
Write the new projection and run it as a new consumer group that starts from the earliest offsets, so it replays every retained event and builds its state. This requires that retention, or compaction, kept the events it needs.
Consumers can receive the same event again after a crash or rebalance, before their offset was committed. Storing the last applied offset or event id alongside the read model, in the same transaction, lets a projection skip events it already applied, so replays don't double-count.
A snapshot stores an aggregate's state as of a given event version. Loading the aggregate then means reading the snapshot plus only the events after it, instead of replaying thousands of events every time a command is handled.