Securing & Governing AI · Chapter 12
Auditing an AI system — what evidence looks like
Read this if you have been asked to audit something nobody can quite explain.
An audit needs evidence: artefacts that exist independently of anyone's assurances, that you can inspect, and that would look different if the control were not operating. AI systems produce perfectly good evidence — it is just unfamiliar, so people do not ask for it and then conclude the system is unauditable.
| Control objective | Evidence to request | What a failing answer sounds like |
|---|---|---|
| Inventoried and owned | Register entry with named owner and tier | “It's part of the platform” |
| Purpose and limits defined | Written intended and prohibited use | “People use it for all sorts of things” |
| Quality is measured | Eval suite, dated golden set, baseline, last N runs | “It performs well”, no artefact |
| Changes controlled | Change log tying versions to eval runs and approvals | Prompts edited in production by anyone |
| Data governed | Data-flow diagram, classification, retention, DPA | No diagram exists |
| Access appropriate | Tool permission list; two-user retrieval test | “It uses a service account” |
| Actions traceable | Tool-call logs: identity, arguments, result, time | Only the final answer is logged |
| Humans in the loop | Approval config plus a sample of real approvals | A policy saying so, no configuration |
| Failures detected | Monitoring thresholds, alerts, incidents closed | “Users would tell us” |
| Explainable after the fact | One reproduced decision, end to end | “It's a black box” |
The single best test
Pick one real output from last month and ask them to reconstruct it: exact input, retrieved context, model version, prompt version, tool calls, output, and who saw it. That one request exercises logging, versioning, retention and traceability at once. If they can do it, most of your evidence set exists. If they cannot, you have found the finding — and you did not need to understand a neural network to find it.
“It's a black box” is a finding, not an answer
The model's internals are opaque; the system is not. Inputs, retrieved documents, versions, tool calls, outputs and approvals are ordinary logs. If none exist, the issue is not that AI is unauditable — it is that this system was built without observability, and that has a straightforward remediation.