What Is Event Streaming? Why Kafka
On this page
The problem with request-response: checkout calls everyone
When an order is placed, the order service calls inventory, then payments, then email, then analytics, then the loyalty program. Each call adds latency. If analytics is slow, checkout is slow. If email is down, checkout fails. Adding a fraud check means changing and redeploying the order service. Every new consumer of "an order was placed" makes the producer more fragile.
Event-driven architecture: publish the fact once
Event-driven architecture turns this around. The order service publishes one event, OrderPlaced, and returns. Inventory, payments, email, analytics, loyalty and next year's fraud check each subscribe and react on their own schedule. The producer doesn't know or care who's listening.
What makes Kafka different: the log
Older message brokers such as RabbitMQ typically delete a message once a consumer acknowledges it. Apache Kafka stores events in an append-only log and keeps them for a configured retention period (days, weeks or forever), whether or not anyone has read them. Each consumer group simply remembers its position in the log.
That one design choice gives you:
- Independent consumers: a slow analytics job never holds up payments; each reads at its own pace.
- Replay: a new service can read the entire history. A bug in a consumer can be fixed and the events reprocessed.
- Buffering: a consumer can be down for hours and catch up when it returns.
- Throughput: sequential disk writes and batching let a modest cluster handle hundreds of thousands to millions of events per second.
Kafka was built at LinkedIn around 2010 to move activity and log data between systems, and became an Apache project in 2011. It's now used for event-driven microservices, data pipelines, change data capture and stream processing.
Worked example: order placement with and without Kafka
Request-response: the order service calls five services in sequence. Total latency is the sum of all five. Availability is roughly the product of all five. Adding a consumer means changing the producer.
With Kafka: the order service writes the order to its database and publishes OrderPlaced to the orders topic, then responds. Each downstream service runs a consumer group on that topic. If email is down for an hour, emails go out an hour late; checkout never notices. A new fraud service starts reading tomorrow, from the beginning of the retained history if it wants.
When request-response is still the right choice
Events don't replace every call. When the caller needs an answer now to continue, "is this card valid?", "how much stock is left?", a synchronous call is right. The common pattern: synchronous for questions that block the user, events for facts other services react to.
What this course covers
Days 2 to 7 cover the core model: brokers, topics, partitions, producers, consumers and delivery guarantees. Days 8 to 14 cover schemas, replication, retention, rebalancing, Connect, transactions and key design. Days 15 to 21 cover stream processing and patterns such as the outbox. Days 22 to 27 cover running Kafka in production.
Check yourself
The producer publishes each fact once to a topic and doesn't know or care who reads it. A new consumer simply subscribes with its own consumer group and starts reading, possibly from the beginning of the retained log. In a request-response design, the producer would have to add a new call for every new interested service.
It keeps the message in the log until retention expires; consumers track their own positions, so others can read it and anyone can replay it.
When it needs the answer right now to decide what to do next, such as authorizing a payment before confirming an order, or checking stock before accepting it. Events suit telling others what happened; queries and decisions that block the user's request still need request-response.
Checkout's latency becomes the sum of all those calls and its availability the product of theirs, so a slow analytics service or an email outage breaks orders. Publishing an OrderPlaced event lets the other services react on their own time while checkout stays fast and available.
Because consumed records aren't deleted, a service added months later can read the history from the beginning to build its own state. With a traditional queue, that history would be gone once the original consumers acknowledged it.