Infrastructure & Reliability · Reliability, operations & platforms · Chapter 4 of 33
Storage, durability, and the restore you have not tested
A backup that has never been restored is a hypothesis. This chapter is about turning it into evidence.
Storage failures are rare and total, which makes them hard to fund and easy to under-test. Every organisation that has lost data had backups running successfully.
The four questions that matter
- RPO — how much data can we lose? Determined by backup frequency and replication lag. Hourly backups mean up to an hour of loss, whatever the retention policy says.
- RTO — how long can we be down? Determined by restore speed, not backup speed. A restore that succeeds in forty hours does not meet a four-hour objective, and that gap is discovered during outages.
- Is it consistent? A database copied without quiescing may restore into a corrupt state. File-level success tells you nothing about that.
- Can it be reached by whatever destroyed production? This is the modern one. If ransomware holds production credentials and those credentials can delete the backups, you do not have backups. Immutable or air-gapped copies are the control.
Durability is not availability
A storage service advertising eleven nines of durability is telling you the data will not be lost. It says nothing about whether you can reach it during an incident. Those are different guarantees and conflating them produces recovery plans that assume an available service in exactly the situation where it is not.
The only honest answer to “are we protected?”
“As confident as the date of our last full restore test, which was [date], and it took [duration].”
If that date is more than six months old, the correct answer is that you do not know — and saying so is far stronger than an assurance you cannot support. Then also test whether someone other than the author can perform the runbook, because at 3am it will not be them.
Choosing storage by access pattern, not by name
| Kind | Semantics you get | Choose when | The trap |
|---|---|---|---|
| Block | A disk. Random read/write, filesystem on top | A database's own files; anything needing real filesystem semantics | Attached to one machine at a time. It is not a sharing mechanism, and treating it as one ends badly |
| File | A shared filesystem with locking and directories | Legacy applications that expect a POSIX filesystem | Latency and cost. Network filesystems are convenient and slow, and the convenience hides the slowness until scale |
| Object | Whole-object PUT and GET, no partial writes, no rename | The default for most data. Cheap, effectively unbounded, and durable | Not a filesystem. A 'directory' is a name prefix, and a 'rename' is a copy plus a delete — which is why listing a large prefix is slow and why so many pipelines are unexpectedly expensive |
The object-storage trap is worth expanding, because entire data platforms are built on the misunderstanding. There are no directories — only keys that share a prefix. Listing is therefore a scan, not a lookup, and gets slower as the prefix grows. Renaming a ‘directory’ means copying every object and deleting the originals, which is why the commit step of a naive batch job can take longer than the computation.
- Durability and availability are different guarantees. Extremely high durability means the bytes will not be lost. It does not mean you can read them right now, and the availability figure is always the lower of the two.
- Consistency has quietly become strong for reads-after-writes at the major providers — a change that invalidated a decade of defensive coding patterns, and many codebases still carry the workarounds.
- Storage classes trade retrieval for price, and the retrieval cost is where people are surprised. Archival tiers can cost more to read once than to store for a year, so a lifecycle policy that moves data before anyone has checked the access pattern is a common and expensive mistake.
- Egress is the invisible line. Storing data is cheap; moving it between regions or out to the internet is frequently the largest bill, and it does not appear in the storage line item where anyone would look for it.
Check yourself