Disaster Recovery and Multi-Region Architecture
On this page
When the whole region goes
Redundancy across instances and zones (Day 76) handles a failed server or even a failed data center. Disaster recovery (DR) handles the bigger events: a whole cloud region unavailable, a ransomware attack, a mistaken script that deletes production data. These are rare, and when they happen, the plan you wrote and tested is all you have.
RTO and RPO: the two numbers behind every DR plan
- RTO (recovery time objective): how long the business can tolerate the system being down. "We must be back within 1 hour."
- RPO (recovery point objective): how much data the business can tolerate losing, measured in time. "We can lose at most 5 minutes of orders."
They come from the business, not from engineering, because each minute you shave off costs money. Ask what an hour of downtime and an hour of lost data cost; that sets how much the DR setup is worth.
Four disaster recovery strategies
These names come from AWS's disaster recovery guidance and are widely used:
| Strategy | What runs in the second region | Typical RTO | Typical RPO | Cost |
|---|---|---|---|---|
| Backup and restore | Nothing; backups are copied there | Hours to a day | Hours (since last backup) | Lowest |
| Pilot light | Data replicated continuously; app servers off | Tens of minutes to hours | Seconds to minutes | Low |
| Warm standby | A scaled-down, fully working copy | Minutes | Seconds | Medium |
| Multi-region active-active | Full capacity, serving live traffic | Near zero | Near zero to seconds | Highest |
Backup and restore: rebuild everything from infrastructure code and restore data from backups. Fine for internal tools; too slow for customer-facing revenue systems.
Pilot light: the database replicates to the second region continuously, but app servers aren't running. In a disaster, start them (infrastructure as code makes this fast), promote the replica, switch DNS.
Warm standby: a smaller working copy runs all the time, which also lets you test it with real requests. In a disaster, scale it up and switch traffic.
Active-active: both regions serve users all the time. Losing one means the other absorbs its traffic. The hard part is data: writes in two regions need either a globally consistent database (Spanner, CockroachDB, Aurora DSQL), conflict handling (Day 67), or partitioning users by home region so each record has one writer.
The dependencies that live in the region you lost
A recovery region is only independent if everything recovery needs is there too. Walk the failover runbook and ask, for every step, where the tool runs:
- Container images and packages: is the registry replicated, or only in the failed region?
- Secrets and certificates: is the key vault or secrets manager available in the second region?
- Identity and DNS: can you log in to the console and change DNS records if the primary region's control services are struggling?
- Infrastructure code and pipelines: can you deploy without the CI system that runs in the region you lost?
The goal is static stability: the recovery region keeps working, or can be started, without changing anything in the failed one. Replicating data is the easy part; this list is where real DR plans fail.
Backups are not optional, even with replicas
Replication copies mistakes instantly (Day 25). Protect against deletion, corruption and ransomware with:
- Point-in-time recovery for databases.
- Copies in another region and another account, so one compromised account can't delete everything.
- Immutable or locked backups that can't be deleted before their retention period ends.
- Regular restore tests. A backup you've never restored is a hope, not a backup.
Test the plan with DR drills
Run a DR drill at least a couple of times a year: really fail over to the second region (or restore from backups into a clean account), time each step, and compare it with your RTO and RPO. Drills always find something: a missing secret, an expired certificate, a DNS TTL of a day, a runbook step that no longer exists.
Check yourself
RTO 4 hours, RPO 15 minutes. Pilot light fits: continuous data replication meets the RPO, and starting app servers fits within the RTO.
Replication copies every change, including deletes, to every replica within seconds, so the bad script's damage reaches the other region too. Recovering needs point-in-time backups you can restore to just before the mistake, ideally immutable and stored in a separate account so the same mistake or attacker can't delete them.
Every dependency of recovery replicated or available in the second region: images, secrets, DNS control and pipelines, tested in the drill.
Handling writes in more than one region: you need a globally consistent database, conflict resolution, or partitioning so each record has a single home region.
Backup and restore, pilot light (core data replicated, everything else started on demand), warm standby (a scaled-down copy running), and multi-site active-active. Each step costs more and shortens recovery time, so choose based on the RTO and RPO the business needs.