Designing a Resilient Network Architecture
On this page
The design exercise
Before reading on, sketch this: a public web application and API with users worldwide. It must survive the loss of a server, an availability zone and, with a short interruption, a whole region. Decide what happens at each layer: DNS, edge, load balancing, compute, data and security.
A resilient design, layer by layer
DNS (Days 11 and 12). A managed DNS service with health-checked failover or latency-based routing between two regions. Records for the regional endpoints use short TTLs (60 seconds) so failover takes effect quickly. Use a DNS provider with anycast and a strong availability record; DNS is a single point of failure for everything else.
Edge (Days 20 and 24). A CDN in front of both regions serves cached content even if an origin is down, absorbs DDoS traffic, terminates TLS near users and offers HTTP/3. Many CDNs can also fail over between origins themselves, faster than DNS can.
Load balancing (Day 22). In each region, a Layer 7 load balancer spread across at least three availability zones, with health checks removing failed instances. Only it, and only from the CDN, accepts traffic.
Compute (Days 18 and 25). Instances or pods across all zones in private subnets, with autoscaling, stateless app servers and sessions in a shared store. Capacity planned so losing one zone leaves enough to serve peak traffic (with three zones, each must handle about half the load, not a third).
Data. A primary database with a synchronous standby in another zone within the region, and asynchronous replication to the second region. Asynchronous means a region failover can lose the last seconds of writes, so decide in advance whether that's acceptable for each kind of data; it's the CAP and PACELC trade-off in practice.
Security (Days 18 to 21). Default-deny security groups between tiers, encryption on every hop, admin access through a session manager, flow logs and alerts.
Connectivity. Redundant NAT gateways (one per zone), private endpoints for cloud services, and no hidden dependency on a single zone or region (a single NAT gateway, a config service in one place, a DNS resolver).
Tracing a region failure
The primary region becomes unreachable:
| Time | What happens |
|---|---|
| 0 s | Health checks from the DNS service and CDN start failing for region A |
| ~30 s | The CDN marks the origin unhealthy and sends misses to region B; cached pages kept serving throughout |
| ~1 to 2 min | DNS failover answers with region B; resolvers pick it up as the 60-second TTL expires |
| Minutes | Region B's database replica is promoted to primary (automatically or by an operator); writes resume |
| Ongoing | Region B autoscales to carry full traffic, which is why it must have headroom or fast scaling |
Losses: requests in flight at the moment of failure, and any writes not yet replicated. Users see a short outage or a read-only period, not an extended failure.
What makes it work in practice
- Test failover regularly. Untested failover usually fails: DNS TTLs ignored by clients, missing capacity, a forgotten dependency in the failed region. Run game days.
- Avoid shared fate: the control plane you need during a failover (deploy tools, DNS management, runbooks) must not live only in the region that failed.
- Match the design to the requirement. Multi-region active-active is expensive and complex. Many systems are well served by multi-zone in one region plus backups restorable in another. Write down recovery time and data loss targets (RTO and RPO) and design to them.
Check yourself
In normal operation each zone carries a third of peak, but when one zone fails, the remaining two must carry all traffic, so each needs about half of peak load. Running each zone at a third is too little headroom; plan for 50 percent each, plus some margin.
Resolvers cache answers for the TTL, so a long TTL keeps users pointed at a failed region for that long. Short TTLs, like 60 seconds, let DNS failover reach most users within a minute or two, at the cost of more DNS queries.
Writes committed in the failed region but not yet replicated, usually the last few seconds, depending on replication lag at that moment. Monitor that lag, since it's your real recovery point.
Untested failover often fails on details like stale DNS, missing capacity or unreplicated secrets. Scheduled drills prove it works and measure recovery time.
It's a single point of failure: if its zone fails, private instances in the other zones lose outbound access. Use one NAT gateway per zone.