Securing & Governing AI · Chapter 15 of 9
The AI supply chain
Where the model came from, what it was built on, and how you would know if any of it changed.
Software supply chain security is a mature discipline: provenance, signing, SBOMs, verified builds. The AI supply chain is the same problem with more links and far less tooling, and most organisations have not mapped it at all.
The links, and what can go wrong at each
| Link | The risk | What to ask |
|---|---|---|
| The base model | You cannot inspect the weights and cannot verify what went into them | Who publishes it, under what licence, and is the version pinned? |
| Model distribution | A model downloaded from a public hub could be substituted or malicious | Are you verifying a checksum or signature, and from where are you pulling? |
| Serialisation format | Some formats execute code on load — a genuine remote-code-execution path | Are models loaded from safetensors rather than pickle-based formats? |
| Fine-tuning data | Poisoned or simply wrong data becomes behaviour you cannot remove | Where did it come from, who reviewed it, is it retained and versioned? |
| Embedding model | Frequently a different vendor, changing it silently invalidates your whole index | Is it versioned, and what happens when it changes? |
| Frameworks and plugins | Ordinary dependency risk, in a fast-moving ecosystem with young projects | Are they in your normal dependency scanning, or outside it? |
| The hosted API | The vendor may update the model beneath you without notice | Is there version pinning and a deprecation notice period in the contract? |
The two that surprise people
Model files can execute code. The historically common serialisation format in the Python ecosystem is pickle-based, and loading a pickle executes arbitrary code by design. A model file from an untrusted source is therefore an executable, not a data file. The mitigation is straightforward — prefer safetensors, scan model files, and pull from a vetted internal mirror rather than directly from a public hub — and it is unfamiliar enough that many teams have never considered it.
The silent vendor update is the one that bites in production. A hosted model can change behaviour on a Tuesday for a system you validated on a Monday, with no code change on your side and no entry in your change log. This breaks the assumption every change-management control rests on.
Make the eval suite the acceptance gate
The control that addresses the silent update is not a contract clause on its own — it is pinning the version, requiring notice of deprecation, and then making a passing run of your evaluation suite the condition for accepting a new version.
That turns an uncontrollable external event into an ordinary change with a test attached, which is a shape your existing governance already knows how to handle.
A minimum viable AI bill of materials
- Base model, version, provider, and licence — licences vary far more than people assume and some restrict commercial use or downstream training.
- Embedding model and version, with a note of what re-indexing costs if it changes.
- Fine-tuning datasets, their provenance, and where they are retained.
- Frameworks and their versions, inside your normal dependency scanning.
- The prompt, versioned in source control, because it is part of the system's behaviour.
- Who owns each of the above.