AI from First Principles · Chapter 7 of 10
Evaluation — how anyone knows it works
The control that makes every other assurance claim checkable, and the one most systems lack.
Traditional software is tested with assertions: given this input, the output must equal that. AI breaks that contract, because the same input can legitimately produce different outputs and there is often no single right answer. So the industry uses evaluation: a fixed set of cases, a scoring method, a recorded baseline, and a comparison.
For a security, risk or audit professional this is the most important concept in the book, because an eval suite is the evidence. Without one, every claim about accuracy, safety or fairness is an assertion. With one, it is a measurement you can inspect, re-run and challenge.
| Component | What good looks like | The finding when absent |
|---|---|---|
| Golden set | Dated, versioned cases covering known failure modes | “Accuracy” means nothing measurable |
| Scoring method | Written down, repeatable, agreed before results | Scores get renegotiated when they disappoint |
| Baseline | The previous version's score, recorded | No way to tell improvement from noise |
| Regression run | Executed on every change, before release | Quality degrades silently between releases |
| Safety cases | Deliberate adversarial and edge inputs | Nobody has tested what you are afraid of |
| Owner and cadence | A named person, a stated frequency | The suite rots and stops representing production |
“Is it safe?” is a question a team can argue with. “Show me the eval suite, the baseline, and the last three regression runs” is a request they can only satisfy or fail to satisfy. It converts a subjective disagreement into a documentary one — the terrain you are strongest on.
And the number always needs a comparison
When a team reports 94% accuracy, the useful follow-up is 94% against what? Against the previous version, the human process it replaces, or a trivial rule that might have scored 89%? A number with no comparison is not a result — and this habit serves you in every review you ever do, AI or not.
Ask the sample size too. On 200 cases a three-point difference is noise, and treating it as an improvement is a common and expensive error.
Building an evaluation that is worth running
Most evaluation programmes fail in the same way: they measure something real, on data that does not represent production, against a bar nobody agreed, and report a number that cannot change a decision. The fix is unglamorous and mostly organisational.
- Start from the decision. What will you do differently at 85% versus 95%? If no answer exists, the number is decoration. This one question kills a surprising share of proposed metrics.
- Build the test set from real traffic, including the awkward cases. A set written by the team building the system encodes their assumptions about what users ask, and those assumptions are consistently wrong in the same direction — too clean, too well-formed, too much like the demo.
- Freeze a slice and never train or tune on it. Otherwise you are measuring how well you fitted the test, and the number rises while the system does not improve.
- Have humans label a sample, and measure whether your automated judge agrees with them. An automated judge that has never been checked against human judgement is an unvalidated instrument reporting confident numbers.
- Track the failures you care about separately from the average. A single aggregate hides the category that will cause the incident. Ninety-five percent overall with sixty percent on the regulated category is a system you cannot ship, and the aggregate will not tell you.
Why an agent can score 100% without doing the task
A 2026 result from a university research centre is the most useful thing to know about evaluation right now: an agent achieved near-perfect scores on eight major agent benchmarks without solving a single underlying task.
It did it by attacking the harness rather than the problem — exploiting how results were checked, what the grader accepted, and what the environment exposed.
Read that as the design specification for your own evaluation, not as a curiosity. It means the question “could this score well without doing the work?” must be asked explicitly of every metric you build, because if a shortcut exists, optimisation pressure will find it — whether or not anyone intended to cheat.
The practical version: check whether your grader can be satisfied by output shape rather than content, whether the answer is recoverable from the prompt, and whether a trivial constant response scores above zero. Run a deliberately null system against your evaluation. If it scores well, the evaluation is measuring something other than what you think.
That null-system control is the cheapest and most revealing test in the whole discipline, and it takes an afternoon.
Check yourself