Lakshya

Infrastructure & Reliability · Chapter 1 of 8

What actually makes a system reliable

Not uptime. A shared, written definition of acceptable, and a budget for spending the rest.

3 min read1 diagramAll 8 chapters

Reliability conversations go wrong because ‘the system should be up’ is not a specification. Engineering will always want more reliability than the business will fund, and the business will always want more features than reliability allows, and without a number that argument recurs every quarter with whoever is loudest winning.

WHAT AN AVAILABILITY TARGET ACTUALLY BUYS YOU, PER MONTH 99% 7h 18m one bad afternoon and you are out99.9% 43m one bad deploy consumes most of it99.99% 4m 19s you cannot fix anything by hand in that time99.999% 26s only achievable if humans are not in the path Each extra nine costs roughly ten times more and removes humans from one more decision. So the question is never “how many nines” — it is what a minute of downtime actually costs, and who pays for it.
Nines are not a preference, they are a budget. Each extra nine costs roughly an order of magnitude more and removes humans from one more decision — at four nines nobody can log in and fix anything within the budget, so recovery must be automatic.

The three terms, kept separate

  • SLI — the measurement. The proportion of requests served successfully under 300ms, say. It must be measured where the user is, not at the server: a request that never arrived is still a failure to them.
  • SLO — the target for that measurement. 99.9% over 28 days. This is an internal commitment and it should be a little stricter than anything you promise externally.
  • SLA — a contract with financial consequences. Most teams do not have one and confuse it with the SLO.

The error budget is the useful part

If the target is 99.9%, then 0.1% of failure is permitted — roughly 43 minutes a month. That is not a tolerance to be minimised; it is a resource to be spent on shipping. A team with budget remaining should be moving faster, and a team that has exhausted it should stop and repair. Written down in advance, that turns a recurring political argument into a rule.

Choose the SLO from user pain, not from what you currently achieve

The most common failure is setting the target at whatever the system happens to do today, which measures nothing and can never be missed in a way that means anything.

The test: would a user notice the difference between meeting and missing this? If not, it is the wrong indicator — and an SLO on a metric nobody feels will be defended in postmortems by people who know it does not matter.

What actually causes outages

Very little of it is hardware. In practice the dominant causes are change — a deploy, a config edit, a certificate expiring, a flag flipped — and dependency failure, where something you rely on degrades rather than dies. Which is why the highest-value reliability work is usually not redundancy; it is staged rollout, fast rollback, and graceful degradation when a dependency goes slow.

← All chaptersCapacity, and the cost of being wrong in each direction →