Catalog My learning Blog
Apache Kafka Tutorial: Event Streaming to Production · Phase 4 · Lesson 26 · 4 min read

Kafka MirrorMaker and Disaster Recovery

On this page
  1. Kafka disaster recovery: when the whole region goes
  2. MirrorMaker 2: replicating between clusters
  3. Why consumer offsets need translation
  4. Active-passive vs active-active Kafka
  5. Kafka failover runbook (active-passive)
  6. Check yourself

Kafka disaster recovery: when the whole region goes

Replication factor 3 across availability zones (Day 9) survives a broker or even a zone failing. It doesn't survive losing the region, a misconfiguration that corrupts the cluster, or a cloud account being compromised. For that you need a second cluster elsewhere, kept up to date. Set the targets first: how much data can you lose (RPO) and how long can you be down (RTO) (System Design Roadmap, Day 77)?

MirrorMaker 2: replicating between clusters

MirrorMaker 2 (MM2), part of Apache Kafka and built on Kafka Connect (Day 12), replicates between clusters:

  • Topic data: records are consumed from the source and produced to the target. Replication is asynchronous, so the target lags a little (your RPO).
  • Topic configs and ACLs: kept in step.
  • Consumer group offsets: translated and synced, so consumers can fail over (below).

By default, replicated topics are prefixed with the source cluster's alias: orders from cluster primary becomes primary.orders on the target, which prevents loops when replicating in both directions. IdentityReplicationPolicy keeps the original names for one-way setups.

Code
# mm2.properties
clusters = primary, dr
primary.bootstrap.servers = kafka-eu-1:9093
dr.bootstrap.servers = kafka-us-1:9093
primary->dr.enabled = true
primary->dr.topics = orders, payments, customers.*
primary->dr.sync.group.offsets.enabled = true
primary->dr.emit.checkpoints.enabled = true
replication.factor = 3
Code
bin/connect-mirror-maker.sh mm2.properties

Managed alternatives include Confluent Cluster Linking and replication features in cloud Kafka services, some of which preserve offsets exactly.

MirrorMaker 2 copies topics and translated consumer offsets from the primary cluster to a disaster-recovery cluster; on failover, consumers resume from the translated offsets.
MirrorMaker 2 copies topics and translated consumer offsets from the primary cluster to a disaster-recovery cluster; on failover, consumers resume from the translated offsets.

Why consumer offsets need translation

The same record usually has a different offset on each cluster: retention may have started at different times, and transactional markers and compaction differ. A consumer that read up to offset 81,000 on the primary can't just start at 81,000 on the DR cluster. MM2 periodically records checkpoints mapping source offsets to target offsets per consumer group, and with sync.group.offsets.enabled writes the translated offsets into the target's __consumer_offsets. Because checkpoints are periodic, failover typically causes a small amount of reprocessing, so consumers must be idempotent (Day 7).

Active-passive vs active-active Kafka

Active-passiveActive-active
Writes go toOne primary clusterBoth clusters, each for local producers
ReplicationPrimary → DRBoth directions (prefixed topic names)
ConsumersOn the primary; DR consumers stopped or read-onlyRead local topics plus the other cluster's prefixed copies
FailoverPromote DR: start producers and consumers thereShift traffic; the other side already serves
ComplexityLowerHigher: two copies of every stream, ordering only per cluster

Most teams start with active-passive.

Kafka failover runbook (active-passive)

  1. Decide to fail over: primary unrecoverable within the RTO.
  2. Stop remaining producers to the primary, if any can still reach it, to avoid split writes.
  3. Check replication lag at the moment of failure: that's the data you may have lost. Record it.
  4. Start consumers on the DR cluster; they pick up translated group offsets and reprocess a little.
  5. Point producers at the DR cluster (DNS or config change), writing to the topic names consumers expect.
  6. Failback later deliberately: replicate DR → primary, then switch back in a quiet window.

Rehearse this. A DR plan that has never run will fail on a detail: a missing ACL on the DR cluster, a consumer configured with the wrong topic prefix, offsets not syncing because the group was active.

Check yourself

MirrorMaker copies records but doesn't preserve offsets, so the same record usually has a different offset on each cluster. MirrorMaker 2 emits checkpoints that map consumer groups' offsets from the source to the target, and consumers must start from those translated offsets after failover.

Replication between clusters is asynchronous, so whatever MirrorMaker hadn't copied yet when the primary failed is lost or stranded. Its replication lag at that moment is your effective recovery point, so monitor and alert on it.

Prefixing replicated topics with the source cluster's alias, such as primary.orders, distinguishes local from remote topics and prevents records being replicated back and forth endlessly in active-active setups. Consumers can subscribe to both with a pattern when needed.

Both sides accept writes, so the same key can be updated in two regions at once and the clusters end up disagreeing. Applications must partition ownership of keys by region or be designed to merge conflicting updates.

Failover involves switching clients' bootstrap servers, translating consumer offsets and checking replication lag, and any of these can silently break as configurations change. A scheduled drill proves the recovery point and recovery time you actually achieve, not the ones on paper.