Infrastructure & Reliability · Reliability, operations & platforms · Chapter 10 of 33
Linux performance: the method, and the six numbers that lie to you
A ten-minute checklist that beats an hour of guessing, and the metrics whose ordinary interpretation is wrong.
Almost every production performance investigation is done by guessing, and almost every one could be done by method. The method is USE — for every resource, check utilisation, saturation and errors — and it converges because it is exhaustive over resources rather than exhaustive over hypotheses.
Six metrics whose obvious reading is wrong
1. Load average is not CPU utilisation. On Linux it counts processes in runnable and in uninterruptible sleep — which means disk I/O wait is in there too. A load average of 40 on an 8-core box can mean CPU saturation, or it can mean forty processes blocked on a slow NFS mount with the CPUs completely idle. The number on its own does not distinguish them, and it is averaged over one, five and fifteen minutes, so it also lags.
2. ‘Free’ memory is not the memory you have. Linux deliberately uses everything spare for page cache, so free approaching zero is normal and healthy. The number to read is available, which estimates what could be reclaimed. Alerting on free memory generates permanent false alarms and teaches everyone to ignore memory alerts, which is worse than having none.
3. Disk %util does not mean what it meant in 2005. It measures the proportion of time at least one request was in flight. On a spinning disk that genuinely meant saturation. On an SSD or an NVMe device serving dozens of requests in parallel, 100% %util can be a device at a fraction of its capability. Read await and the average queue size instead.
4. CPU steal time is somebody else's problem. %st in top is time your virtual CPU was ready to run and the hypervisor gave the physical core to another tenant. You cannot fix it in your application. Persistent steal is a signal to move instance type or provider, and recognising it saves days of misdirected optimisation.
5. A high context-switch rate is not automatically bad — but the ratio matters. /proc/PID/status separates voluntary switches (the thread blocked, waiting for I/O or a lock) from involuntary ones (the scheduler preempted it). High involuntary means CPU contention. High voluntary means the thread is waiting for something, and the something is your actual problem.
6. An average latency is close to useless. If 1% of requests take ten seconds and 99% take 10 ms, the mean is 110 ms and nobody experiences 110 ms. Always read the distribution. And note that percentiles do not compose — you cannot average p99s across hosts and get the fleet p99.
Memory: the part that produces the most confusing incidents
Linux memory accounting is where good engineers make wrong calls, because the abstractions are genuinely subtle.
- RSS is not what the process costs. Resident set size counts shared pages — shared libraries, copy-on-write pages after a fork — once per process. Sum the RSS of ten forked workers and you will substantially over-count.
PSS, proportional set size, divides shared pages among their users and is the honest number. - Overcommit means
mallocrarely fails. Linux hands out address space optimistically and only allocates physical pages on first touch. So a program that allocates far more than exists will run happily until it writes to those pages, at which point the OOM killer arrives. This is why memory exhaustion presents as a sudden death rather than as a graceful allocation failure. - The OOM killer's choice is scoreable and tunable. It picks by
oom_score, weighted mostly by memory footprint, adjustable per process withoom_score_adj. The classic production surprise is that the largest process is usually the important one — so the database gets killed and the log shipper survives. - Swap is not obsolete, and disabling it is usually wrong. A little swap lets the kernel evict genuinely cold anonymous pages and keep more page cache, which is a good trade. What you must not tolerate is sustained swap traffic — watch
siandsoinvmstat, not the amount of swap used. Used swap is history; swapping now is the emergency. - In a container, the limit that kills you is the cgroup's, not the node's. A process can be OOM-killed on a machine with plenty of free memory because its own cgroup hit its limit. Read
memory.maxandmemory.current, notfree.
cgroups, and the CPU throttling that looks like a network problem
This is the single most valuable thing in this chapter for anyone running containers, because the symptom looks nothing like the cause.
A CPU limit in Kubernetes becomes a CFS bandwidth quota: the cgroup is granted a quota of CPU time per 100 ms period, and when it is exhausted every thread in the cgroup is stopped until the next period begins. Not slowed — stopped.
So a service with a 500 millicore limit that needs 60 ms of CPU to serve a request will sometimes be frozen for 50 ms mid-request. The p99 latency becomes appalling while average CPU utilisation reads a comfortable 30%, because the freezes are invisible to a utilisation dashboard. Teams chase the network, the database and the load balancer for weeks.
How to find it, and what to do
Find it: cat /sys/fs/cgroup/cpu.stat and read nr_throttled and throttled_usec. In Prometheus it is container_cpu_cfs_throttled_seconds_total. Any sustained throttling on a latency-sensitive service is a finding.
What to do: raise the limit, or remove it. The increasingly common practice is to set CPU requests and no CPU limit — requests guarantee a share under contention, which is what you actually wanted, while limits cap you even when the node is idle, which is almost never what you wanted.
Memory is the opposite. Always set a memory limit, because memory is not compressible — a runaway process with no limit takes the node down and everything on it. Limit memory, request CPU is a good default and a good sentence to be able to justify.
The tools, in escalating order
| Level | Tool | What it gives you |
|---|---|---|
| Overview | vmstat 1, mpstat -P ALL 1, iostat -xz 1 | The USE table in three commands. Start here, always |
| Per-process | pidstat -t 1, top -H | Per-thread. A process at 100% of one core on a 16-core box is a single-threaded bottleneck, and the process view hides that |
| Syscalls | strace -c -p PID | Which syscalls, how many, how long. Enormous overhead — never on a hot production process |
| Profiling | perf record -F 99 -p PID -g → flame graph | Where CPU time actually goes. The single highest-value tool here |
| Kernel tracing | eBPF — bpftrace, biolatency, execsnoop | Low overhead, production-safe, answers questions nothing else can |
| Off-CPU | off-CPU flame graphs | Where the thread was blocked. Most latency problems are off-CPU, and standard profiling cannot see them at all |
That last row deserves emphasis. Ordinary profiling samples running threads, so it shows you where CPU is spent. But a slow request is usually slow because a thread was waiting — on a lock, a disk, a downstream call. Off-CPU analysis is the technique that finds it, and knowing that on-CPU profiling structurally cannot answer the question is a strong signal.
Check yourself