Infrastructure & Reliability · Reliability, operations & platforms · Chapter 2 of 33
Capacity, and the cost of being wrong in each direction
Under-provision and you fail. Over-provision and you fund nothing else. The arithmetic is learnable in an afternoon.
Capacity planning has an asymmetry that decides how you approach it: being 30% under is an outage, and being 30% over is money that could have been headcount. Neither is free, so the goal is a defensible number with stated assumptions rather than a comfortable one.
The ladder
Every capacity answer is the same chain, and interviewers for infrastructure roles want to hear you walk it out loud with numbers:
- Start from the given — requests per day, users, transactions.
- Convert to peak, never average. A day is about 86,400 seconds, and real traffic peaks at two to three times its mean. Systems fail at peak.
- Per-unit capacity — what one node serves at a sustainable utilisation. Measure it if you can; state the assumption if you cannot.
- Divide. This is the number most people stop at, and it is the answer to a question nobody operates.
- Multiply for failure domain. Survive losing a node, and survive losing a zone. That is usually a 1.5× on the raw number.
- Multiply for headroom — growth, and the fact that autoscaling is not instant. Then say the assumption list out loud.
The constraint is rarely CPU
Name the binding constraint explicitly. In practice it is usually connections, memory, IOPS, a downstream rate limit, or a licence — and if you cannot name which, you have not finished the design. The corollary is that scaling the obvious tier frequently changes nothing, because the queue simply moves.
Little's Law, the one piece of theory worth carrying
Concurrency = arrival rate × latency. If you serve 500 requests a second and each takes 200ms, you have 100 in flight at any moment — which tells you your connection pool, your thread count and your memory footprint.
It also explains the death spiral: latency rises, so concurrency rises, so resources exhaust, so latency rises further. Systems do not degrade gracefully through that unless you make them — with load shedding, which is admitting you would rather fail 5% of requests fast than fail all of them slowly.
Why systems fall over a cliff instead of degrading
Capacity planning is usually taught as arithmetic — measure usage, add headroom, extrapolate. That misses the property that actually causes outages: a system's response to load is not linear, and near saturation it is close to vertical.
The queueing result behind it is worth internalising even without the mathematics. As utilisation rises, waiting time rises in proportion to 1/(1−utilisation). At 50% utilisation, queueing roughly doubles the service time. At 90% it multiplies it by ten. At 99% by a hundred. The step from 90% to 95% utilisation doubles your latency while looking like a five-point change on a dashboard.
- This is why the answer to “how much headroom?” is not a percentage of average load — it is however much keeps you off the steep part of that curve at peak.
- It is why averages hide outages. A service averaging 40% utilisation can be at 95% for three minutes every hour, and those three minutes are your incident. Plan against the peak percentile, not the mean.
- And it is why adding capacity sometimes fixes a problem that looks like a code bug. A slow endpoint under saturation is usually a queueing symptom rather than a slow endpoint.
- Little's Law is the other free result: concurrency equals arrival rate times latency. It means that when latency rises, in-flight work rises with it — so a slow dependency silently consumes your thread pool or connection pool, and the failure surfaces somewhere unrelated to the actual cause.
The failure mode this predicts is the cascade, and it is the one that turns an incident into an outage.
A dependency slows down. Your latency rises, so your in-flight requests rise, so you exhaust the connection pool. Now every request fails, including ones that never touched the slow dependency. Callers retry, which adds load, which makes it worse.
Retries are the accelerant. Naive retry turns a degraded dependency into a denial of service against it, and every layer that retries multiplies the effect — three layers with three retries each is twenty-seven times the load at the bottom.
The controls are unglamorous and they work: timeouts shorter than your caller's timeout; bounded concurrency per dependency, so one slow thing cannot consume the whole pool; a circuit breaker that stops calling a failing dependency; exponential backoff with jitter, because synchronised retries reconverge into a thundering herd; and load shedding, which is the deliberate choice to fail some requests fast rather than fail all of them slowly.
The question that finds whether these exist: “what is the timeout on this call, and what happens when it fires?” A surprising number of production calls have no timeout at all, which means the answer is ‘wait forever’.
Check yourself