Securing & Governing AI · Chapter 12 of 13
Auditing an AI system — what evidence looks like
Read this if you have been asked to audit something nobody can quite explain.
An audit needs evidence: artefacts that exist independently of anyone's assurances, that you can inspect, and that would look different if the control were not operating. AI systems produce perfectly good evidence — it is just unfamiliar, so people do not ask for it and then conclude the system is unauditable.
| Control objective | Evidence to request | What a failing answer sounds like |
|---|---|---|
| Inventoried and owned | Register entry with named owner and tier | “It's part of the platform” |
| Purpose and limits defined | Written intended and prohibited use | “People use it for all sorts of things” |
| Quality is measured | Eval suite, dated golden set, baseline, last N runs | “It performs well”, no artefact |
| Changes controlled | Change log tying versions to eval runs and approvals | Prompts edited in production by anyone |
| Data governed | Data-flow diagram, classification, retention, DPA | No diagram exists |
| Access appropriate | Tool permission list; two-user retrieval test | “It uses a service account” |
| Actions traceable | Tool-call logs: identity, arguments, result, time | Only the final answer is logged |
| Humans in the loop | Approval config plus a sample of real approvals | A policy saying so, no configuration |
| Failures detected | Monitoring thresholds, alerts, incidents closed | “Users would tell us” |
| Explainable after the fact | One reproduced decision, end to end | “It's a black box” |
The single best test
Pick one real output from last month and ask them to reconstruct it: exact input, retrieved context, model version, prompt version, tool calls, output, and who saw it. That one request exercises logging, versioning, retention and traceability at once. If they can do it, most of your evidence set exists. If they cannot, you have found the finding — and you did not need to understand a neural network to find it.
“It's a black box” is a finding, not an answer
The model's internals are opaque; the system is not. Inputs, retrieved documents, versions, tool calls, outputs and approvals are ordinary logs. If none exist, the issue is not that AI is unauditable — it is that this system was built without observability, and that has a straightforward remediation.
What evidence actually looks like
“We have AI governance” is unfalsifiable. An audit turns it into a set of artefacts someone can inspect, and knowing which artefacts are asked for — and which are easy to produce only if you were doing the work — is the whole skill.
| Claim | Weak evidence | Evidence that holds |
|---|---|---|
| "We know where AI is used" | A policy requiring registration | The inventory itself, with dates, owners and tier — plus a discovery mechanism that finds unregistered use, because self-registration finds only the compliant |
| "We evaluate our models" | A results slide | The frozen test set, the harness, results over time, and the record of a version change re-run against it |
| "Humans review the output" | A workflow diagram | The override rate. If reviewers approve 99.8% of suggestions, that is rubber-stamping — and you can measure it without asking anyone |
| "We handle data properly" | A data-protection assessment | What happens on a deletion request, traced end to end — including whether the data reached a fine-tuned model, where deletion means retraining |
| "We monitor for drift" | A dashboard | An alert that fired, and what was done about it. A dashboard nobody has ever acted on is a screensaver |
The override rate is the single most revealing metric in AI assurance, and it is cheap to compute.
Human oversight is the control organisations lean on hardest — and it is the one most often theatre. A reviewer facing a queue of confident-sounding suggestions, under time pressure, with no signal about which are likely wrong, will approve almost all of them.
An override rate near zero does not mean the model is excellent. It means the control is not operating, and it is indistinguishable from having no review at all while looking, on paper, like a strong safeguard.
What makes oversight real: the reviewer can see why the suggestion was made, they have enough time and enough context to disagree, disagreeing is not penalised, and there is a route for the disagreement to change the system rather than just this case.
And the honest test, which almost nothing passes: “show me a case where the human disagreed and the outcome changed.” If none exists, the oversight is decorative, and you have found that out before an incident rather than after.
Check yourself