Most production pain reduces to three things, and each has a known treatment that organisations consistently under-apply.

• Stage everything. Canary, then a percentage, then the fleet, with an automatic halt condition. Global simultaneous deployment is a design choice, and the 2024 CrowdStrike and 2021 Fastly incidents are what it costs when it is the wrong one.
• Make rollback the default response, not a decision requiring approval. If rolling back needs a meeting, it will not happen fast enough.
• Test the rollback, because a rollback path nobody has exercised is a plan, not a control. This is especially true where a schema change is involved and the rollback is not symmetric.
• Consistency matters as much as correctness. Knight Capital lost roughly $440m in 45 minutes in 2012 because a deployment reached seven of eight servers and an old flag meant something new on the eighth.

Mitigate before you understand. Restoring service and diagnosing cause are different activities and the first has priority — rolling back without knowing why is usually correct. Declare early, because the cost of an unnecessary incident call is small and the cost of a late one is not. And separate roles: someone commanding, someone communicating, someone investigating. One person doing all three does none well.

Toil is manual, repetitive work that scales with the system and produces no lasting value. Left alone it grows until there is no capacity for anything else, and it is invisible because it always looks like the urgent thing.

The treatment is to cap it — a stated fraction of time, commonly 50%, reserved for engineering — and to measure it honestly. And to notice that a runbook is a step on the way to automation, not a destination: a well-written runbook for a recurring task is a specification for the script that should replace it.