AI from First Principles · Chapter 7
Evaluation — how anyone knows it works
The control that makes every other assurance claim checkable, and the one most systems lack.
Traditional software is tested with assertions: given this input, the output must equal that. AI breaks that contract, because the same input can legitimately produce different outputs and there is often no single right answer. So the industry uses evaluation: a fixed set of cases, a scoring method, a recorded baseline, and a comparison.
For a security, risk or audit professional this is the most important concept in the book, because an eval suite is the evidence. Without one, every claim about accuracy, safety or fairness is an assertion. With one, it is a measurement you can inspect, re-run and challenge.
| Component | What good looks like | The finding when absent |
|---|---|---|
| Golden set | Dated, versioned cases covering known failure modes | “Accuracy” means nothing measurable |
| Scoring method | Written down, repeatable, agreed before results | Scores get renegotiated when they disappoint |
| Baseline | The previous version's score, recorded | No way to tell improvement from noise |
| Regression run | Executed on every change, before release | Quality degrades silently between releases |
| Safety cases | Deliberate adversarial and edge inputs | Nobody has tested what you are afraid of |
| Owner and cadence | A named person, a stated frequency | The suite rots and stops representing production |
“Is it safe?” is a question a team can argue with. “Show me the eval suite, the baseline, and the last three regression runs” is a request they can only satisfy or fail to satisfy. It converts a subjective disagreement into a documentary one — the terrain you are strongest on.
And the number always needs a comparison
When a team reports 94% accuracy, the useful follow-up is 94% against what? Against the previous version, the human process it replaces, or a trivial rule that might have scored 89%? A number with no comparison is not a result — and this habit serves you in every review you ever do, AI or not.
Ask the sample size too. On 200 cases a three-point difference is noise, and treating it as an improvement is a common and expensive error.