Infrastructure & Reliability · Reliability, operations & platforms · Chapter 8 of 33
Chaos engineering and game days
Controlled experiments on a running system, the human rehearsal that finds a different class of problem, and how to start without frightening anyone.
Chapter 1 defined what reliable means and gave you a budget. This is how you find out whether the thing you built actually behaves the way you think it does — before an incident tells you it does not.
The name is wrong, and it matters
Chaos engineering is not breaking things randomly. It is the opposite: a controlled experiment with a written hypothesis, one variable, a bounded blast radius and an abort condition. The name conjures a monkey loose in production, which is the single biggest obstacle to getting it approved — and the obstacle is entirely linguistic.
The framing that gets it approved
You are trying to disprove a hypothesis, not to cause an outage. The hypothesis is that your system already handles this. If nothing breaks, the experiment succeeded — you now have evidence rather than an assumption.
The failure you are injecting is going to happen anyway. A node will die, a dependency will slow down, a disk will fill. The only choice is whether it happens on a Tuesday afternoon with the team watching, or at 3am on a bank holiday.
Put that second sentence in the proposal. It reframes the ask from ‘may we break production’ to ‘may we choose when this happens’, and that is a different conversation.
Steady state is the whole discipline
Before you inject anything you must say what ‘working’ looks like as a number you already collect. Not ‘the service is up’ — something like orders completed per minute stays within its normal band, or p99 checkout latency stays under two seconds.
It must be a business-visible signal, not a component one. CPU is not steady state; a healthy pod count is not steady state. Both can look perfect while customers cannot buy anything — which is precisely the failure mode you are hunting.
And if the team cannot produce that number, stop there. You have found the real gap, it cost you nothing, and fixing it makes everything downstream possible. A surprising number of chaos programmes deliver their best finding in the first meeting.
What to inject
Why latency beats failure, and it is the most useful thing in this chapter.
Systems are usually built to handle a dependency being DOWN. They are rarely built to handle it being SLOW.
A dead dependency refuses the connection immediately, your call fails fast, your fallback runs. A dependency answering in eight seconds instead of eighty milliseconds holds every one of your threads open while it does it — and your connection pool drains, and requests that have nothing to do with that dependency start queueing, and you go down because something else got slow.
That cascade is the single most common shape of a real large-scale outage, and injecting a few hundred milliseconds of latency is the cheapest way to find out whether you have it.
What experiments actually find
- Retry storms. Every caller retries, nobody added jitter, and the moment the struggling service recovers it is flattened by the backlog. The system's own recovery behaviour is what prevents recovery.
- Timeouts that do not compose. The classic: a caller with a 3-second timeout in front of a service whose own retry budget is 10 seconds. The caller gives up and retries while the first attempt is still running, so load triples under stress.
- A fallback that has never executed. Written two years ago, never exercised, and it throws on the first call because a field was renamed.
- A hidden dependency. The service you were told was standalone turns out to call the identity provider on every request.
- And the best possible outcome: an alert that did not fire. The system degraded exactly as predicted, nobody was told, and you found that out on a Tuesday. Every experiment should check what alerted, not only what broke.
Game days — the same idea pointed at people
A game day runs a realistic scenario against real systems with the actual humans, in real time. Chaos experiments test whether the system copes. Game days test whether the organisation does — and they reliably find a different class of problem.
| Chaos experiment | Game day | |
|---|---|---|
| Tests | The system | The people and the process |
| Typically finds | Retry storms, timeouts, missing fallbacks | Stale runbooks, unclear ownership, missing access |
| Runs | Automated, continuously | Scheduled, a few times a year |
| Needs | A tool and an abort switch | Calendars, which is the harder problem |
The findings from a first game day are almost never technical, and they are almost always the same three. The runbook references a dashboard that was migrated. The one person who knows how the failover works is on leave, and nobody else has the access. And the recovery procedure requires a tool that authenticates through the single sign-on which is part of the scenario.
That last one recurs so often it is worth designing against directly: at least one recovery path must not depend on the systems most likely to be down. See operational resilience, where the same problem appears in its most severe form — recovering when the identity provider itself is compromised.
How to start without frightening anyone
- 1. Run a game day first. No production risk, no tooling, no approval battle — a room, a scenario and two hours. It will produce findings, which is what buys you permission for step 2.
- 2. Experiment in staging, and be honest that staging lacks the traffic, the data volume and the people. It proves the mechanism, not the behaviour.
- 3. Production, off-peak, smallest dose, abort armed, everyone watching. One instance, one percent of traffic, a few hundred milliseconds.
- 4. Widen only after a boring result. If the experiment found something, fix it and re-run the same one before escalating scope.
- 5. Automate last. Continuous automated chaos is the mature end state and a terrible beginning — an unattended experiment that finds something real at 2am is just an outage you caused.
Two things to fix before you start, not after: your observability has to be good enough to see the effect, or you are injecting failure blind — and you need a written rule that an experiment is never run during an active incident or a change freeze, because someone will otherwise do it during one.
What it does not tell you
It tests what you thought of. An experiment is a hypothesis, and you cannot hypothesise about a failure mode you have never imagined — so a long clean run is evidence about your imagination as much as about your system.
Which is why the regulatory cousin exists. Scenario testing under an operational resilience programme runs end to end, including suppliers and people, against a scenario somebody else chose. Chaos gives you continuous evidence that known failure modes are handled; scenario testing gives you periodic evidence that the business service survives a bad day. Most organisations have one and not the other, and they are usually run by teams that never speak — see operational resilience.
Check yourself