Infrastructure & Reliability · Reliability, operations & platforms · Chapter 7 of 33
Running production
Change, incidents and toil. The three things that consume an operations team, and what to do about each.
Most production pain reduces to three things, and each has a known treatment that organisations consistently under-apply.
Change is the dominant cause of outages
- Stage everything. Canary, then a percentage, then the fleet, with an automatic halt condition. Global simultaneous deployment is a design choice, and the 2024 CrowdStrike and 2021 Fastly incidents are what it costs when it is the wrong one.
- Make rollback the default response, not a decision requiring approval. If rolling back needs a meeting, it will not happen fast enough.
- Test the rollback, because a rollback path nobody has exercised is a plan, not a control. This is especially true where a schema change is involved and the rollback is not symmetric.
- Consistency matters as much as correctness. Knight Capital lost roughly $440m in 45 minutes in 2012 because a deployment reached seven of eight servers and an old flag meant something new on the eighth.
Incidents: the decisions are the content
Mitigate before you understand. Restoring service and diagnosing cause are different activities and the first has priority — rolling back without knowing why is usually correct. Declare early, because the cost of an unnecessary incident call is small and the cost of a late one is not. And separate roles: someone commanding, someone communicating, someone investigating. One person doing all three does none well.
Toil is the thing that quietly consumes the team
Toil is manual, repetitive work that scales with the system and produces no lasting value. Left alone it grows until there is no capacity for anything else, and it is invisible because it always looks like the urgent thing.
The treatment is to cap it — a stated fraction of time, commonly 50%, reserved for engineering — and to measure it honestly. And to notice that a runbook is a step on the way to automation, not a destination: a well-written runbook for a recurring task is a specification for the script that should replace it.
The question that finds an unhealthy operations team
“What fraction of your time last month was project work rather than reactive?”
If nobody has the number, that is the finding. If it is under about 20%, the team is not improving anything and the situation will compound — because the only route out is engineering time they no longer have.
On-call that people can sustain
An on-call rotation is a system with its own failure modes, and the commonest one is not technical: the rotation degrades until the good engineers leave, and the alerts that caused it are still firing.
| Symptom | What it actually means | The fix |
|---|---|---|
| Alerts nobody acts on | The alert encodes a condition with no action attached | Delete it. An alert with no runbook and no action is a notification, and it trains people to ignore the channel |
| Pages outside working hours that could wait | Severity is being assigned by the alert author, not by user impact | Two tiers: page for user-visible harm now, ticket for everything else. Enforce it |
| Same incident every week | Toil is being absorbed rather than fixed | Cap it: a recurring page becomes engineering work with a named owner after the second occurrence |
| Only one person can fix it | A single point of failure with a pulse | Rotate ownership deliberately, and treat a runbook only that person could follow as unwritten |
| Nobody writes postmortems | They are being read as blame, or nothing changes as a result | Assign and track the actions. A postmortem whose actions are never done is a ritual, and everyone can tell |
The measurement that tells you whether on-call is healthy
Not the number of incidents. The number of pages per on-call shift that required a human to do something only a human could do.
Everything else is either an alert that should be a ticket, a condition that should have been automated, or a recurring failure that should have been fixed. Counting it separately makes the distinction visible and makes it arguable.
The threshold most teams find sustainable is roughly two or fewer per week, with none most nights. Above that, quality of response falls — people stop investigating and start restarting things — and the strongest engineers quietly begin looking elsewhere.
The related discipline: an alert must name the user impact and link a runbook. ‘CPU above 80%’ is not an alert, because CPU being high is not itself a problem. ‘Checkout latency above two seconds for five minutes’ is, because someone is having a bad time right now and there is something to do about it.
And publish the on-call load alongside delivery metrics. An organisation that measures velocity and not toil will keep trading one for the other without ever seeing the trade.
Check yourself