Catalog My learning Blog
Event-Driven Architecture Course: Event Sourcing and CQRS · Phase 1 · Lesson 3 · 8 min read

Message Brokers and Pub/Sub in Event-Driven Systems

On this page
  1. Learning Objective
  2. Concept Explanation
  3. Worked Example: An Idempotent Consumer over At-Least-Once Delivery
  4. Real-World Examples
  5. Review Questions & Answers

Learning Objective

Explain how a broker decouples producers from consumers, contrast the queue model with the durable log model, and describe the delivery guarantees, ordering constraints, and idempotency requirements that every event-driven consumer has to be designed around.

Concept Explanation

A broker is the piece of infrastructure that makes decoupling real rather than aspirational. Without one, a producer publishing an event would still have to know where to send it. With one, the producer writes to a named destination, a topic or an exchange, and the broker takes responsibility for durability, delivery, buffering, and fan-out. The broker is also what converts a consumer outage from a data-loss event into a latency event: the messages pile up, and when the consumer returns it works through the backlog.

There are two broad families, and the difference matters more than vendor choice. A traditional message queue, the model behind RabbitMQ, ActiveMQ, and Amazon SQS, treats a message as work to be handed out: a consumer receives it, acknowledges it, and the broker deletes it. Delivery is competitive, each message goes to exactly one member of a consumer group, and once everyone has acknowledged, the message is gone. A durable log, the model behind Apache Kafka, Apache Pulsar, and Amazon Kinesis, treats messages as an ordered, append-only sequence retained for a configured period regardless of who has read it. Consumers track their own position, the offset, and reading does not consume. That single property, that reading does not destroy, is what makes replay possible, and replay is the foundation of the event sourcing and projection-rebuild techniques that the rest of this course depends on.

Ordering is where most designs quietly go wrong. Brokers do not generally guarantee global ordering across an entire topic, because global ordering would mean a single writer and no horizontal scale. What they guarantee is ordering within a partition, a shard, or a single queue. Kafka assigns a message to a partition by hashing its key, so all events sharing a key, say all events for order 9f12, land in the same partition and are consumed in the order they were written, while events for different orders may be interleaved arbitrarily. Choosing the partition key is therefore a correctness decision, not a performance tuning knob: it defines the exact scope within which your consumers can rely on ordering.

Delivery guarantees come in three flavors, and only two of them are practical. At-most-once delivery acknowledges before processing, so a crash loses the message; almost nobody wants this. At-least-once delivery acknowledges after processing, so a crash between processing and acknowledgment causes redelivery, and the consumer sees the same event twice. Exactly-once delivery across arbitrary systems is not achievable in the general case, because acknowledging the message and committing the side effect are two different systems and cannot be made atomic without a distributed transaction. What modern brokers offer instead is either exactly-once processing within their own boundary, as Kafka does with idempotent producers and transactional writes across topic partitions and consumer offsets, or the practical alternative: at-least-once delivery plus an idempotent consumer, which produces the same observable result as exactly-once without needing the impossible guarantee.

Idempotency is therefore not an optimization but a baseline requirement. Every event carries a unique event id; every consumer records the ids it has already processed, in the same transaction as its side effect, and skips duplicates. When the side effect is in an external system that has no such table, a bank transfer, a third-party charge, the standard technique is an idempotency key passed to that system so it deduplicates on its own side. The rest of the operational vocabulary follows from these same realities: consumer groups for scaling out, dead letter queues for messages that fail repeatedly so a poison message cannot block a partition forever, and backpressure or consumer lag monitoring so a consumer falling permanently behind is detected before the retention window silently drops data it never read.

PropertyQueue model (RabbitMQ, SQS)Durable log model (Kafka, Pulsar, Kinesis)
Message lifetimeDeleted once acknowledgedRetained for a configured time or size, regardless of reads
ConsumptionDestructive - one consumer takes the messageNon-destructive - each consumer group tracks its own offset
ReplayNot possible once acknowledgedReset the offset and reprocess from any retained point
Ordering scopePer queue, lost when multiple consumers competePer partition, determined by the partition key
Adding a new consumerSees only messages published from now onCan start from the beginning of retention and catch up
Natural fitTask distribution and work queuesEvent streams, projections, and audit logs

Worked Example: An Idempotent Consumer over At-Least-Once Delivery

The consumer below assumes it will see duplicates, because with at-least-once delivery it certainly will. The deduplication insert and the business side effect share one database transaction, which is what makes the guarantee hold across a crash at any point.

JavaScript
// Table that makes the consumer idempotent. The primary key is the event id,
// so a duplicate insert fails and tells us we have already handled this event.
CREATE TABLE processed_events (
  event_id    UUID PRIMARY KEY,
  consumer    TEXT NOT NULL,
  processed_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

async function handleOrderPlaced(event) {
  const tx = await db.begin();
  try {
    // Both the dedup marker and the side effect commit together, or neither does.
    await tx.query(
      'INSERT INTO processed_events (event_id, consumer) VALUES ($1, $2)',
      [event.eventId, 'loyalty-service'],
    );
  } catch (err) {
    if (err.code === '23505') {       // unique_violation - already processed
      await tx.rollback();
      await broker.ack(event);        // safe: the effect happened on the first pass
      return;
    }
    throw err;
  }

  await tx.query(
    'UPDATE loyalty_balances SET points = points + $1 WHERE customer_id = $2',
    [Math.floor(event.totalCents / 100), event.customerId],
  );
  await tx.commit();
  await broker.ack(event);            // crash before this line -> redelivery -> dedup catches it
}
A broker sitting between one producer and three consumer groups, showing partitioned ordering by key, per-group offsets in the durable log model, and the at-least-once redelivery path that an idempotent consumer absorbs using a processed-events table.
A broker sitting between one producer and three consumer groups, showing partitioned ordering by key, per-group offsets in the durable log model, and the at-least-once redelivery path that an idempotent consumer absorbs using a processed-events table.

Real-World Examples

  • Apache Kafka introduced idempotent producers and transactions in version 0.11.0 in 2017, giving exactly-once processing semantics within Kafka itself by atomically committing produced messages together with consumer offsets. Confluent documented clearly at the time that this guarantee holds for Kafka-to-Kafka stream processing and does not extend to arbitrary external side effects, which is exactly why consumers that write to an external database still need their own idempotency mechanism.
  • Stripe's API has supported idempotency keys since 2015 and documents them prominently for exactly the reason described here: a client that times out on a charge request cannot tell whether the charge happened, so it retries, and without a deduplication key on the server side the customer would be charged twice. It is the same problem an at-least-once event consumer faces, solved the same way, and it is the standard reference implementation teams cite when designing their own idempotent handlers.
Best PracticeDesign every consumer to be idempotent from day one, and record the deduplication marker in the same transaction as the side effect. At-least-once delivery means duplicates are not an edge case to be handled later; they are a normal, expected occurrence that will happen during every broker rebalance, every consumer restart, and every network blip.

Review Questions & Answers

Ordering is guaranteed only within a partition, and the key determines which partition a message lands in. If order events are keyed by orderId, every event for a given order is processed in the order it was produced, which is usually what the business logic requires. If they are keyed by something coarser like region, ordering per order is still preserved because all of an order's events still share a partition, but throughput may be skewed. If they are keyed randomly or not at all, OrderCancelled can be processed before OrderPlaced, and the consumer's state machine will see an impossible sequence. The key defines the exact scope in which your correctness assumptions hold.

Because acknowledging the message to the broker and committing the write to the database are operations on two independent systems, and there is no way to make them atomic without a distributed transaction protocol that both systems support and that adds its own availability cost. Whichever order you choose, there is a window in which a crash leaves the two out of step: acknowledge first and you can lose the write, commit first and you can be redelivered. The practical answer is at-least-once delivery plus an idempotent consumer, which makes the duplicate harmless and therefore produces exactly-once observable behavior without the impossible guarantee.

A durable log retains messages after they are read, so a consumer can reset its position and reprocess history, and a brand-new consumer can start from the beginning and catch up on everything that happened before it existed. A queue deletes on acknowledgment, so once a message is consumed it is gone. That replay capability is exactly what event sourcing depends on to rebuild state from an event stream, and what CQRS depends on to rebuild or add a read-model projection from scratch without asking producers to resend anything.

Retries handle transient failures, a brief network partition, a database failover, a temporarily overloaded dependency, where the same message will succeed on a later attempt. They do not handle a poison message: one that will fail forever because the payload is malformed, references a deleted entity, or triggers a bug. Without a dead letter queue, that message is retried indefinitely and, in an ordered partition, blocks every message behind it, turning one bad record into a full consumer outage. The dead letter queue moves the message aside after a bounded number of attempts so processing continues, and preserves it for human inspection instead of discarding it.