Security Engineering · Chapter 11 of 12
Operational resilience
Not uptime — the discipline of staying inside a limit when something actively breaks you, and the regulatory frame that now demands you prove it.
The reliability chapter is about keeping systems up. This is a different question, and increasingly a regulated one: when something breaks you — a failure, a supplier, or an attacker — can the business service keep working well enough, for long enough, that customers are not seriously harmed?
The distinction is not pedantry. Reliability is measured in averages over periods and owned by engineering. Resilience is measured in single events against a hard ceiling, and owned by the board.
The three ideas the regulators built this on
Important business service · impact tolerance · severe but plausible
An important business service is described from the customer's side. “Make a payment.” “Get a claim assessed.” “Trade a security.” Not ‘the payments platform’ — that is a system, and systems are how you deliver the service, not what it is.
An impact tolerance is the maximum disruption to that service before the harm becomes intolerable — to customers, to the market, to the firm's viability. Expressed in time, and set at board level.
A severe but plausible scenario is the test condition. Not worst-case — nobody plans for an asteroid — but explicitly not the comfortable case either. Your primary data centre is gone. Your main cloud region is unavailable for a day. Ransomware has encrypted your identity provider. A critical supplier is offline for a week.
The obligation is to demonstrate you stay inside the tolerance under those scenarios — by testing, not by assertion. That single word, demonstrate, is what turns this from a plan into a programme.
Why step 2 is where programmes stall
To set a tolerance you must first know what the service actually depends on, and almost nobody does. The mapping runs through systems you own, systems another team owns, a vendor, that vendor's own cloud provider, a certificate someone renews manually, and a person who knows how to restart the batch.
The mapping exercise is universally described as tedious and universally reported as the most valuable part — because the findings are not subtle. The single point of failure nobody knew about is usually found on a whiteboard, not in an incident.
- Concentration risk becomes visible. Six services that look independent turn out to share one identity provider, one message broker, or one supplier — and the tolerance for each was set as though the others would survive.
- Fourth-party risk appears. Your supplier's resilience is only as good as their supplier's, and their contract with you probably says nothing about it.
- The manual step surfaces. Nearly every map contains at least one recovery path that requires a named individual to be reachable.
Testing it, and the two traditions that should meet
| Chaos engineering | Regulatory scenario testing | |
|---|---|---|
| Comes from | Engineering practice | Supervisory expectation |
| Asks | “What happens if this component fails?” | “Can the SERVICE stay inside its tolerance?” |
| Scope | One dependency, often in production | End to end, including people and suppliers |
| Cadence | Continuous, automated | Periodic, evidenced, reported |
| Blind spot | Tests what you thought of | Too infrequent to catch drift |
They are complementary and are usually run by people who never speak. Chaos experiments give you continuous evidence that a failure mode is handled; scenario tests give you evidence that the business service survives a bad day including the human parts. A resilience programme with only the second is a slide deck; one with only the first cannot answer the question a regulator asks.
Game days sit between them — a real scenario, run against real systems, with the actual people. The reliable finding from a first game day is not technical: it is that the runbook assumes a tool that requires the single sign-on which is part of the scenario.
Recovery, and the question that changes the answer
Recovery targets are usually stated as RTO — how quickly you must be back — and RPO, how much data you can afford to lose. Both are ordinary continuity planning. The security scenario breaks the assumptions underneath them.
Conventional disaster recovery assumes the failure was an accident. Ransomware is not an accident, and it invalidates three assumptions at once.
Your backups are a target. Modern intrusions delete or encrypt backups first, deliberately, because that is what converts an outage into a payment. A backup reachable with production credentials is not a backup. Immutable, and credentially separated, or it does not count.
Failover may replicate the problem. A hot standby faithfully receives the encryption. Geographic redundancy answers a fire, not an adversary.
And the one that surprises people: what do you restore INTO? If the identity provider is compromised, you cannot authenticate to the tools you need to recover. Rebuilding trust — a clean directory, clean credentials, a known-good network — is the long pole in every serious ransomware recovery, and it is almost never in the DR plan.
Test the restore, not the backup. A backup job reporting success is evidence that a job ran. Only a restore is evidence that you have data.
Where the obligation comes from
The regulatory shape is worth knowing even outside finance, because it is spreading. The European Union's DORA regime, applying since January 2025, regulates technology resilience directly rather than reaching it through financial reporting — and notably reaches through to critical technology suppliers rather than stopping at the regulated firm's own walls. India's revised RBI Directions take the same posture, with resilience named in the title. Contrast that with older regimes like SOX, which only touch IT because financial numbers happen to flow through it — see risk and controls.
The practical consequence for an engineer: resilience stopped being an internal quality argument you have to win on merit, and became something with an external deadline and an examiner. That is genuinely useful leverage for work that was always worth doing and never won a prioritisation meeting.
Check yourself