Infrastructure & Reliability · Reliability, operations & platforms · Chapter 6 of 33
Debugging that transfers
One method, applied to every resource. It turns troubleshooting from intuition into a checklist you can hand to someone else.
The most valuable skill in infrastructure is not knowing a particular system — it is having a method that works on a system you have never seen. The USE method is that: for every resource, check utilisation, saturation and errors.
Applying it
| Resource | Utilisation | Saturation | Errors |
|---|---|---|---|
| CPU | percent busy | run queue length, pressure stall | throttling |
| Memory | used | swapping, reclaim, OOM kills | ECC errors |
| Disk | percent busy | queue depth, await | I/O errors, retries |
| Network | bits per second | interface discards | CRC errors, drops |
The column that matters is saturation, and it is the one almost never on the default dashboard.
Two habits worth more than any tool
- Diff against a working comparator. If identical hosts exist and one is slow, mechanically compare everything — firmware, kernel parameters, negotiated link speed, memory population. The difference is nearly always in that diff, and finding it that way is much faster than reasoning about it.
- Reason from the timing signature. Failing at a consistent elapsed time suggests something thermal or accumulating; failing at a consistent iteration suggests code; failing at a consistent time of day suggests a scheduled job or a neighbour.
Debugging production, when you did not write it
The production debugging skill is not knowledge of the system — you often have none — it is a search strategy that halves the space with each step, and a refusal to guess before the evidence supports it.
- Start with the change log, not the code. Deploys, config flips, feature flags, dependency upgrades, certificate expiries, scheduled jobs, and traffic. A timestamp plus a deploy list resolves a large share of incidents before anyone opens an editor.
- Establish the blast radius immediately. One user, one tenant, one region, one instance, or everyone? That single answer eliminates most hypotheses at once — a single-region failure is not a code bug, and a single-instance failure is not a deploy.
- Work the request path in order — client, DNS, load balancer, gateway, service, dependency, datastore — and confirm each hop rather than assuming it. Most wasted incident time is spent debugging a layer that was fine.
- Prefer evidence you can gather without a deploy. Logs, metrics, traces, a direct call to the service, a query against the datastore. Anything requiring a release adds twenty minutes and a risk.
- Write the timeline as you go. Not for the postmortem — for you. Ten minutes into an incident nobody can remember what they have already ruled out, and the commonest waste is checking the same thing twice while the real cause sits unexamined.
Correlation, causation, and the deploy that was not the cause
The most common incident-response error is stopping at the first plausible correlation. A deploy went out at 14:15, the errors started at 14:20, so the deploy is the cause. Roll it back.
Often correct. And when it is wrong it is expensive, because the rollback appears to fix things — the real cause was a traffic pattern that also subsided — and the actual defect stays in the system with a false exoneration attached to it.
The cheap discipline: before accepting the correlation, ask what else changed in that window, and whether the failure mode is one this change could plausibly produce. A configuration change to a logging library is unlikely to cause a database timeout, however well the timestamps line up.
Roll back anyway. Restoring service is not the same as diagnosis, and it should not wait for one. But record that the diagnosis is unconfirmed, because a rollback that resolves the symptom is very often taken as proof and closes the investigation.
The signature of this error: the same incident recurring weeks later with a different deploy blamed.
Check yourself