Securing & Governing AI · Chapter 10 of 13
Data governance — the actual hard problem
Not glamorous, and the source of most real AI incidents to date.
Almost every AI data incident that has actually happened, as opposed to been imagined in a workshop, reduces to one of three failures. None involves the model behaving badly.
Failure 1 — the permission model does not survive ingestion
Documents live in systems with per-user access. Somebody embeds them all into one vector index. The index has one permission: whoever can query it. The HR folder, the board pack and the salary review are now answerable to anyone who asks nicely. Nobody attacked anything — the access model was dropped in transit. The control: filter retrieval by the asking user's entitlements at query time, and test it with two accounts of different privilege.
Failure 2 — prompts and outputs become an unmanaged data store
Every prompt is a document, frequently containing customer data, landing in application logs, vendor logs, traces, analytics and debugging tools — none classified for it, most with longer retention than the source, several third parties. The control: classify the prompt and response stream deliberately, decide retention explicitly, redact before logging, and put the AI vendor on the data-flow map.
Failure 3 — the copy nobody tracked
Building AI features generates copies: an export to fine-tune, a sample for evaluation, a snapshot on a laptop, production data in a lower environment. The eval golden set is especially overlooked — by definition a curated collection of your most interesting real cases, usually living in a repository. The control: treat AI project datasets as in-scope from day one, and ask specifically about the eval set, because nobody volunteers it.
Three questions that find most of it
1. “Draw me the data flow from source system to model provider and back, including logs.” The gaps in the drawing are the findings. 2. “Which copies of production data exist for this project, including test and eval sets?” 3. “If we terminated the vendor tomorrow, what would they still hold and for how long?”
The questions that decide whether AI data governance is real
- “What happens to a prompt after we send it?” Is it retained, for how long, is it used for training, who at the provider can see it, and what does the contract actually say — as opposed to what the marketing page says. This is the first question and it is frequently unanswered.
- “Can a user reach data through the model that they could not reach directly?” If retrieval happens before permission filtering, the answer is yes — and the leak can arrive through a summary even when the source document is withheld.
- “What happens on a deletion request?” Trace it. Backups, analytical copies, retrieval indexes, logs of prompts and completions, and anything that reached a fine-tuned model — where deletion means retraining rather than a delete statement. The answer is usually incomplete in the same three places.
- “Where does inference physically happen?” A residency commitment is not met by a provider being available in a region; it is met by the request being processed there, which is a different guarantee and a different contract clause.
- “Are prompts and completions logged, and who can read the log?” Prompt logs are among the most sensitive stores an organisation will create, because users paste things into a chat box they would never put in a ticket — and the log is usually created for debugging, by an engineer, with no classification applied.
The prompt log is the sleeper risk, and it is created by accident.
Users treat a chat interface as private in a way they do not treat a form. Customer names, medical details, credentials, salary figures and entire confidential documents get pasted in, because the interface invites it.
The log capturing that is typically built for debugging, retained indefinitely by default, readable by the engineering team, outside whatever classification scheme the organisation applies to its real data stores, and absent from the data map.
Treat it as the sensitive store it is: classify it, restrict it, set a retention period and enforce it, and include it in the deletion path from day one rather than discovering it during a subject access request.
And a control most teams can implement cheaply: redact obvious identifiers before the log is written, rather than after. Redacting on read protects nothing, because the sensitive value is already at rest.
Check yourself