Infrastructure & Reliability · Chapter 2 of 8
Capacity, and the cost of being wrong in each direction
Under-provision and you fail. Over-provision and you fund nothing else. The arithmetic is learnable in an afternoon.
Capacity planning has an asymmetry that decides how you approach it: being 30% under is an outage, and being 30% over is money that could have been headcount. Neither is free, so the goal is a defensible number with stated assumptions rather than a comfortable one.
The ladder
Every capacity answer is the same chain, and interviewers for infrastructure roles want to hear you walk it out loud with numbers:
- Start from the given — requests per day, users, transactions.
- Convert to peak, never average. A day is about 86,400 seconds, and real traffic peaks at two to three times its mean. Systems fail at peak.
- Per-unit capacity — what one node serves at a sustainable utilisation. Measure it if you can; state the assumption if you cannot.
- Divide. This is the number most people stop at, and it is the answer to a question nobody operates.
- Multiply for failure domain. Survive losing a node, and survive losing a zone. That is usually a 1.5× on the raw number.
- Multiply for headroom — growth, and the fact that autoscaling is not instant. Then say the assumption list out loud.
The constraint is rarely CPU
Name the binding constraint explicitly. In practice it is usually connections, memory, IOPS, a downstream rate limit, or a licence — and if you cannot name which, you have not finished the design. The corollary is that scaling the obvious tier frequently changes nothing, because the queue simply moves.
Little's Law, the one piece of theory worth carrying
Concurrency = arrival rate × latency. If you serve 500 requests a second and each takes 200ms, you have 100 in flight at any moment — which tells you your connection pool, your thread count and your memory footprint.
It also explains the death spiral: latency rises, so concurrency rises, so resources exhaust, so latency rises further. Systems do not degrade gracefully through that unless you make them — with load shedding, which is admitting you would rather fail 5% of requests fast than fail all of them slowly.