Chaos Engineering Principles
On this page
The failover nobody had tested
The architecture diagram shows two database replicas, a circuit breaker on every dependency and automatic failover. Then the primary database really fails, and failover takes 14 minutes instead of 30 seconds because a connection string was cached for the life of the process. The design was right; the system didn't match the design. Chaos engineering finds that gap on a Tuesday afternoon instead of during a real outage.
It's the practice of deliberately injecting failure into a system, in a controlled way, to see whether it behaves as you believe. Netflix popularized it with Chaos Monkey (2011), which randomly terminated production instances so engineers built services that tolerate it.
The chaos engineering method
It's an experiment, not random breakage:
- Define steady state. A measurable signal of normal health that users care about: checkout success rate, orders per minute, p99 latency. Not CPU.
- Form a hypothesis. "If one of the three API instances is terminated, checkout success stays above 99.5 percent and p99 stays under 800 ms."
- Inject the failure, with the smallest blast radius that tests the hypothesis: one instance, one dependency, a small share of traffic.
- Observe steady-state metrics against the hypothesis. Have an abort condition and a way to stop immediately.
- Learn. If the hypothesis held, confidence is now evidence. If not, you found a weakness cheaply. Fix it and run the experiment again.
Common chaos experiments
| Experiment | Tests |
|---|---|
| Terminate an instance | Statelessness, health checks, autoscaling (Days 71, 76) |
| Add 500 ms latency to a dependency | Timeouts, bulkheads, fallbacks (Days 56, 75) |
| Make a dependency return errors | Retries, circuit breakers (Days 56, 74) |
| Fail over the database primary | Promotion, redirection, connection handling (Day 76) |
| Block a whole availability zone | Zone redundancy, capacity headroom |
| Fill a disk, exhaust connections | Alerts, graceful degradation |
Tools: Gremlin, AWS Fault Injection Service, Azure Chaos Studio, Chaos Mesh and LitmusChaos for Kubernetes, and service meshes (Day 57) that can inject delays and errors per route.
How to start chaos engineering safely
- Start in staging, with production-like traffic if possible, then move carefully to production, where the real configuration lives.
- Run during business hours with the owning team watching, never at 3 a.m. unattended.
- Begin tiny: one instance, 1 percent of traffic, a few minutes.
- Automate the stop: abort when the steady-state metric crosses a threshold.
- Game days: a scheduled session where a team runs a larger scenario, such as losing a zone, and practices the response, including the runbooks and communication.
- Have the basics first. Chaos in production needs good dashboards, alerts tied to the SLO (Day 78) and an on-call team that knows the system. Without those, you can't tell what an experiment did, and you're just causing outages.
- Don't experiment during incidents, freezes or peak events, or while the error budget is already spent.
Write up every experiment, even the ones that passed: the hypothesis, what happened and what changed. Over a year, that record is evidence of how the system really behaves, and new team members learn more from it than from the architecture diagram.
Chaos engineering vs testing
Tests check that code does what it should under known conditions. Chaos experiments check how the whole running system behaves under conditions that are hard to reproduce in tests: real configuration, real timeouts, real dependencies, real traffic. Both are needed.
Check yourself
It describes the servers, not the users. CPU can stay low while every checkout fails, or spike while users notice nothing. A chaos experiment needs a steady state that reflects user experience, such as checkout success rate, latency percentiles or orders per minute, so you can see whether the fault actually hurt anyone.
Build the observability first. Without a steady-state signal and alerts, they can't judge an experiment's effect or stop it in time.
Inject failure into the smallest scope that tests the hypothesis, such as one instance or a small share of traffic, with a fast way to stop.
You've learned that the protection for the recommendations dependency, its timeout, bulkhead or fallback, doesn't work as designed, so a slow optional service takes down a critical page. Fix the protection, for example a short timeout with an empty fallback, then rerun the experiment to confirm product pages stay fast.
Tests check that known behavior works under expected conditions. Chaos experiments inject real-world faults, such as killed instances, latency or lost dependencies, into a running system to discover unknown weaknesses, starting with a hypothesis about how the system should behave.