Securing & Governing AI · Chapter 11
Model risk management
Banking has governed consequential models for a decade. Most of it transfers directly.
If you work in financial services you have model risk management already, and AI mostly fits inside it. If you do not, the discipline is worth borrowing wholesale — it is the most mature framework anyone has for “we rely on something we cannot fully explain.”
The core idea is three lines: the team that builds the model, an independent function that validates it, and audit that checks both did their job. The validator must be able to reach a different conclusion from the builder and must not report to them.
What transfers unchanged
- An inventory. You cannot govern models you cannot list. Usually the single most valuable artefact, and most organisations do not have one for AI.
- Tiering by consequence. A meeting-notes summariser and a loan decision do not get the same scrutiny. Tier on impact of being wrong, not technical sophistication.
- Independent validation proportionate to tier, before deployment.
- Ongoing monitoring with thresholds and an owner, because performance drifts as the world moves away from the data.
- Documented limitations — what it must not be used for, written down and communicated to users.
| Classical model risk | What changes with generative AI |
|---|---|
| One output per input | Many valid outputs; validate distributions and use eval suites |
| A clear performance metric | Often no single metric; a rubric plus human review |
| You built it, so you know it | A third-party model you cannot inspect — TPRM becomes central |
| Stable until retrained | The vendor may update beneath you — require notice, re-run evals |
| Bounded input space | Open-ended natural language — adversarial testing is mandatory |
The vendor-update clause worth insisting on
A silently updated model can change behaviour on Tuesday for a system you validated on Monday. Ask for version pinning, advance notice of deprecation, and a documented change process. Then make re-running your eval suite the trigger for accepting a new version.
That is a contract term and a control at once — the most useful single thing an assurance function can add to a third-party AI arrangement.