Catalog My learning Blog
System Design Roadmap: 100-Day Course for Engineers · Phase 8 · Lesson 79 · 4 min read

Chaos Engineering Principles

On this page
  1. The failover nobody had tested
  2. The chaos engineering method
  3. Common chaos experiments
  4. How to start chaos engineering safely
  5. Chaos engineering vs testing
  6. Check yourself

The failover nobody had tested

The architecture diagram shows two database replicas, a circuit breaker on every dependency and automatic failover. Then the primary database really fails, and failover takes 14 minutes instead of 30 seconds because a connection string was cached for the life of the process. The design was right; the system didn't match the design. Chaos engineering finds that gap on a Tuesday afternoon instead of during a real outage.

It's the practice of deliberately injecting failure into a system, in a controlled way, to see whether it behaves as you believe. Netflix popularized it with Chaos Monkey (2011), which randomly terminated production instances so engineers built services that tolerate it.

The chaos engineering method

It's an experiment, not random breakage:

  1. Define steady state. A measurable signal of normal health that users care about: checkout success rate, orders per minute, p99 latency. Not CPU.
  2. Form a hypothesis. "If one of the three API instances is terminated, checkout success stays above 99.5 percent and p99 stays under 800 ms."
  3. Inject the failure, with the smallest blast radius that tests the hypothesis: one instance, one dependency, a small share of traffic.
  4. Observe steady-state metrics against the hypothesis. Have an abort condition and a way to stop immediately.
  5. Learn. If the hypothesis held, confidence is now evidence. If not, you found a weakness cheaply. Fix it and run the experiment again.
The chaos engineering loop: define steady state, form a hypothesis, run a controlled experiment with a small blast radius, then learn and fix.
The chaos engineering loop: define steady state, form a hypothesis, run a controlled experiment with a small blast radius, then learn and fix.

Common chaos experiments

ExperimentTests
Terminate an instanceStatelessness, health checks, autoscaling (Days 71, 76)
Add 500 ms latency to a dependencyTimeouts, bulkheads, fallbacks (Days 56, 75)
Make a dependency return errorsRetries, circuit breakers (Days 56, 74)
Fail over the database primaryPromotion, redirection, connection handling (Day 76)
Block a whole availability zoneZone redundancy, capacity headroom
Fill a disk, exhaust connectionsAlerts, graceful degradation

Tools: Gremlin, AWS Fault Injection Service, Azure Chaos Studio, Chaos Mesh and LitmusChaos for Kubernetes, and service meshes (Day 57) that can inject delays and errors per route.

How to start chaos engineering safely

  • Start in staging, with production-like traffic if possible, then move carefully to production, where the real configuration lives.
  • Run during business hours with the owning team watching, never at 3 a.m. unattended.
  • Begin tiny: one instance, 1 percent of traffic, a few minutes.
  • Automate the stop: abort when the steady-state metric crosses a threshold.
  • Game days: a scheduled session where a team runs a larger scenario, such as losing a zone, and practices the response, including the runbooks and communication.
  • Have the basics first. Chaos in production needs good dashboards, alerts tied to the SLO (Day 78) and an on-call team that knows the system. Without those, you can't tell what an experiment did, and you're just causing outages.
  • Don't experiment during incidents, freezes or peak events, or while the error budget is already spent.

Write up every experiment, even the ones that passed: the hypothesis, what happened and what changed. Over a year, that record is evidence of how the system really behaves, and new team members learn more from it than from the architecture diagram.

Chaos engineering vs testing

Tests check that code does what it should under known conditions. Chaos experiments check how the whole running system behaves under conditions that are hard to reproduce in tests: real configuration, real timeouts, real dependencies, real traffic. Both are needed.

Check yourself

It describes the servers, not the users. CPU can stay low while every checkout fails, or spike while users notice nothing. A chaos experiment needs a steady state that reflects user experience, such as checkout success rate, latency percentiles or orders per minute, so you can see whether the fault actually hurt anyone.

Build the observability first. Without a steady-state signal and alerts, they can't judge an experiment's effect or stop it in time.

Inject failure into the smallest scope that tests the hypothesis, such as one instance or a small share of traffic, with a fast way to stop.

You've learned that the protection for the recommendations dependency, its timeout, bulkhead or fallback, doesn't work as designed, so a slow optional service takes down a critical page. Fix the protection, for example a short timeout with an empty fallback, then rerun the experiment to confirm product pages stay fast.

Tests check that known behavior works under expected conditions. Chaos experiments inject real-world faults, such as killed instances, latency or lost dependencies, into a running system to discover unknown weaknesses, starting with a hypothesis about how the system should behave.