Lakshya

Infrastructure & Reliability · Chapter 6 of 8

Debugging that transfers

One method, applied to every resource. It turns troubleshooting from intuition into a checklist you can hand to someone else.

3 min read1 diagramAll 8 chapters

The most valuable skill in infrastructure is not knowing a particular system — it is having a method that works on a system you have never seen. The USE method is that: for every resource, check utilisation, saturation and errors.

THE SAME HOST, TWO MEASUREMENTS UTILISATION — what the dashboard shows 40% busy — “plenty of headroom” SATURATION — what users feel run queue: processes waiting for a CPU that is already taken 8 waiting — every request queues behind another Utilisation is an average. Saturation is the queue. Slowness lives in the queue. Measure utilisation, saturation and errors for every resource — the USE method — and check saturation first.
Utilisation is an average; saturation is the queue. A host at 40% CPU with eight processes waiting is saturated, and every dashboard will report it as idle. This single distinction resolves a large share of “it is slow but nothing looks wrong” investigations.

Applying it

ResourceUtilisationSaturationErrors
CPUpercent busyrun queue length, pressure stallthrottling
Memoryusedswapping, reclaim, OOM killsECC errors
Diskpercent busyqueue depth, awaitI/O errors, retries
Networkbits per secondinterface discardsCRC errors, drops

The column that matters is saturation, and it is the one almost never on the default dashboard.

Two habits worth more than any tool

  • Diff against a working comparator. If identical hosts exist and one is slow, mechanically compare everything — firmware, kernel parameters, negotiated link speed, memory population. The difference is nearly always in that diff, and finding it that way is much faster than reasoning about it.
  • Reason from the timing signature. Failing at a consistent elapsed time suggests something thermal or accumulating; failing at a consistent iteration suggests code; failing at a consistent time of day suggests a scheduled job or a neighbour.
← The physical layer, and why AI broke the datacentreRunning production →