Securing & Governing AI · Chapter 11 of 13
Model risk management
Banking has governed consequential models for a decade. Most of it transfers directly.
If you work in financial services you have model risk management already, and AI mostly fits inside it. If you do not, the discipline is worth borrowing wholesale — it is the most mature framework anyone has for “we rely on something we cannot fully explain.”
The core idea is three lines: the team that builds the model, an independent function that validates it, and audit that checks both did their job. The validator must be able to reach a different conclusion from the builder and must not report to them.
What transfers unchanged
- An inventory. You cannot govern models you cannot list. Usually the single most valuable artefact, and most organisations do not have one for AI.
- Tiering by consequence. A meeting-notes summariser and a loan decision do not get the same scrutiny. Tier on impact of being wrong, not technical sophistication.
- Independent validation proportionate to tier, before deployment.
- Ongoing monitoring with thresholds and an owner, because performance drifts as the world moves away from the data.
- Documented limitations — what it must not be used for, written down and communicated to users.
| Classical model risk | What changes with generative AI |
|---|---|
| One output per input | Many valid outputs; validate distributions and use eval suites |
| A clear performance metric | Often no single metric; a rubric plus human review |
| You built it, so you know it | A third-party model you cannot inspect — TPRM becomes central |
| Stable until retrained | The vendor may update beneath you — require notice, re-run evals |
| Bounded input space | Open-ended natural language — adversarial testing is mandatory |
The vendor-update clause worth insisting on
A silently updated model can change behaviour on Tuesday for a system you validated on Monday. Ask for version pinning, advance notice of deprecation, and a documented change process. Then make re-running your eval suite the trigger for accepting a new version.
That is a contract term and a control at once — the most useful single thing an assurance function can add to a third-party AI arrangement.
Borrowing from a discipline that already solved most of this
Financial institutions have run formal model risk management for over a decade, and the framework transfers to AI systems better than anything invented for the purpose — because it was designed for exactly this situation: a quantitative artefact whose output drives decisions, whose internals are not fully interpretable, and which can be confidently wrong.
| The classical practice | What it means for an AI system |
|---|---|
| A model inventory | Start here. Most organisations cannot list where AI is making or influencing decisions — including features embedded in purchased software, which is where the unknown exposure usually sits |
| Tiering by materiality | Not every model needs the same scrutiny. Tier by consequence of being wrong, not by technical sophistication — a simple model deciding credit outranks a sophisticated one suggesting tags |
| Independent validation | Someone who did not build it checks it. The single most valuable and most resisted control, because it is slow and it finds things |
| Documented limitations | What the model is not for, written down and attached to it. This is the artefact that prevents the most common real-world failure: correct model, wrong use |
| Ongoing monitoring | Performance drifts because the world changes. A model validated once is a model validated against a world that no longer exists |
| Change control | Sharper for AI, because the model can change under you when a vendor updates it — without a release on your side |
The two risks the classical framework did not have to handle
1. The model can change without your involvement. A hosted model is a dependency you do not version. A provider updates it, and behaviour your evaluation established last quarter may no longer hold — with no deployment on your side to trigger a review.
The controls: pin a model version where the provider offers it, keep a regression suite you re-run on every version change, and negotiate notice of deprecation into the contract. The third is the one procurement can actually get you and rarely asks for.
2. Correct model, wrong use. A model validated for one purpose gets pointed at another because it is available and it seems to work. This is the most common real failure in practice and it is not a technical failure at all — nothing is broken.
Which is why documented limitations matter more than accuracy figures. An accuracy number without a stated scope invites exactly this: someone reads 94%, does not read the population it was measured on, and applies it to a different one.
The governance question that catches it: “what population was this validated on, and is the population it is being used on the same?” Asked routinely, it finds more real exposure than any amount of red-teaming.
Check yourself