Securing & Governing AI · Chapter 16 of 9
Incident response for AI systems
Your existing process mostly works. Four things about it do not, and they are worth deciding before you need them.
Most of incident response transfers unchanged: detection, triage, containment, eradication, recovery, review. What differs is specific and worth working out in advance rather than at two in the morning.
What is different
- You may not be able to reproduce it. A non-deterministic system may not produce the same output twice, so ‘reproduce, then fix’ can fail at the first step. This is why the input, retrieved context, model version and prompt version must all be logged — if you cannot replay it, you cannot investigate it.
- Containment options are unfamiliar. For an ordinary service you roll back. For an AI feature the equivalents are: revert to the previous prompt version, revert to the previous model version, disable the specific tool, tighten a filter, or fall back to the non-AI path. Decide which of those you can do in under five minutes, before you need to. Most teams discover during the incident that reverting a prompt is a deploy.
- Scope is harder to establish. ‘Which other users got a bad answer?’ requires either logged outputs or the ability to replay inputs. If you log only that a request occurred, you cannot answer the question the regulator or the customer will ask.
- The output may be the incident. With a data breach the harm is exposure. With an AI system the harm is frequently something it said — an incorrect commitment, a harmful instruction, a disclosure — and that has legal weight, as the Air Canada ruling established.
The AI-specific incident classes
| Class | What it looks like | First containment move |
|---|---|---|
| Prompt injection | Unexpected tool calls, egress to an unfamiliar destination, output that ignores the system prompt | Disable the affected tool; do not rely on filtering |
| Data exposure through retrieval | A user receives content they are not entitled to | Disable retrieval for the affected corpus; then check whether permissions were ever enforced at query time |
| Quality regression | Complaints rise; evals may or may not show it | Revert the prompt or model version to the last known-good pair |
| Cost runaway | Spend spikes; usually a retry loop | Rate limit and cap; then find the loop |
| Harmful or incorrect output at scale | A pattern of bad answers rather than one | Fall back to the non-AI path while you assess |
The tabletop worth running once
Take one scenario — an assistant that emailed a customer something incorrect and damaging — and walk it through with the people who would actually be involved. The questions that expose the gaps are always the same, and they are almost never answerable the first time:
- How did we find out, and how long after it happened?
- Can we say which other customers received something similar?
- Can we reproduce the exact output, including the retrieved context and versions?
- What is our fastest containment, and has anyone timed it?
- Who decides whether to disclose, and on what threshold?
- What do we tell the customer about why it happened, given we cannot fully explain the model?
The last question is the one to prepare an answer for
“We do not fully understand our own system” is true of the model's internals and is a very poor thing to say to a customer or a regulator.
The answer that works is about the system rather than the model: what the system was permitted to do, what evidence exists of what it did, what control failed, and what has changed since. That is an answerable, auditable account — and it is exactly the evidence set from chapter 12, which is why building it before an incident is worth so much more than building it after.