Infrastructure & Reliability · Chapter 4 of 8
Storage, durability, and the restore you have not tested
A backup that has never been restored is a hypothesis. This chapter is about turning it into evidence.
Storage failures are rare and total, which makes them hard to fund and easy to under-test. Every organisation that has lost data had backups running successfully.
The four questions that matter
- RPO — how much data can we lose? Determined by backup frequency and replication lag. Hourly backups mean up to an hour of loss, whatever the retention policy says.
- RTO — how long can we be down? Determined by restore speed, not backup speed. A restore that succeeds in forty hours does not meet a four-hour objective, and that gap is discovered during outages.
- Is it consistent? A database copied without quiescing may restore into a corrupt state. File-level success tells you nothing about that.
- Can it be reached by whatever destroyed production? This is the modern one. If ransomware holds production credentials and those credentials can delete the backups, you do not have backups. Immutable or air-gapped copies are the control.
Durability is not availability
A storage service advertising eleven nines of durability is telling you the data will not be lost. It says nothing about whether you can reach it during an incident. Those are different guarantees and conflating them produces recovery plans that assume an available service in exactly the situation where it is not.
The only honest answer to “are we protected?”
“As confident as the date of our last full restore test, which was [date], and it took [duration].”
If that date is more than six months old, the correct answer is that you do not know — and saying so is far stronger than an assurance you cannot support. Then also test whether someone other than the author can perform the runbook, because at 3am it will not be them.