Securing & Governing AI · Chapter 10
Data governance — the actual hard problem
Not glamorous, and the source of most real AI incidents to date.
Almost every AI data incident that has actually happened, as opposed to been imagined in a workshop, reduces to one of three failures. None involves the model behaving badly.
Failure 1 — the permission model does not survive ingestion
Documents live in systems with per-user access. Somebody embeds them all into one vector index. The index has one permission: whoever can query it. The HR folder, the board pack and the salary review are now answerable to anyone who asks nicely. Nobody attacked anything — the access model was dropped in transit. The control: filter retrieval by the asking user's entitlements at query time, and test it with two accounts of different privilege.
Failure 2 — prompts and outputs become an unmanaged data store
Every prompt is a document, frequently containing customer data, landing in application logs, vendor logs, traces, analytics and debugging tools — none classified for it, most with longer retention than the source, several third parties. The control: classify the prompt and response stream deliberately, decide retention explicitly, redact before logging, and put the AI vendor on the data-flow map.
Failure 3 — the copy nobody tracked
Building AI features generates copies: an export to fine-tune, a sample for evaluation, a snapshot on a laptop, production data in a lower environment. The eval golden set is especially overlooked — by definition a curated collection of your most interesting real cases, usually living in a repository. The control: treat AI project datasets as in-scope from day one, and ask specifically about the eval set, because nobody volunteers it.
Three questions that find most of it
1. “Draw me the data flow from source system to model provider and back, including logs.” The gaps in the drawing are the findings. 2. “Which copies of production data exist for this project, including test and eval sets?” 3. “If we terminated the vendor tomorrow, what would they still hold and for how long?”