Securing & Governing AI · Chapter 19 of 13
Measuring harm, not just accuracy
The evaluation chapter has a sibling: quality metrics say nothing about who is hurt when the system is wrong.
Chapter 7 covered whether a system works. This one covers whether it harms — a different question with different methods, and the one regulators and internal audit are increasingly asking about.
Four harm classes worth naming separately
| Class | What it looks like | How you test for it |
|---|---|---|
| Allocative | The system distributes something — credit, a shortlist, a price — unevenly across groups | Disaggregated outcome rates, and the false-positive/false-negative split per group |
| Quality of service | It simply works less well for some people — accents, names, dialects, image types | Disaggregated accuracy on a deliberately diverse evaluation set |
| Representational | Outputs stereotype, erase or demean a group | Adversarial prompting plus human review; automated metrics are weak here |
| Safety | Advice that is dangerous if followed — medical, legal, financial, self-harm | Red-team suites of deliberately dangerous prompts, reviewed by domain experts |
The construction problem nobody warns you about
To measure disaggregated performance you need group labels — and collecting protected attributes is itself a privacy and legal question, sometimes prohibited. This is a genuine tension and the honest answers are limited: use proxy datasets built for the purpose, use aggregated or inferred group statistics with stated error, or measure on a consented research panel. What you should not do is conclude that because you cannot measure it, it is not happening.
The question that finds unmeasured harm in five minutes
“Show me your evaluation results broken down by user segment.”
In most organisations the answer is that they do not have that breakdown — which means an aggregate of 94% could be hiding 71% for one group, and nobody would know. That absence is the finding, and it is usually a week of work to fix rather than a research programme.
Measuring harm rather than accuracy
An accuracy figure is an average over a population, and harm is not distributed like an average. A system that is 95% accurate overall can be 60% accurate for one group, and the aggregate will never say so.
- Always disaggregate. Report performance by every group you can legitimately measure. The aggregate is the number that hides the finding, and reporting it alone is the most common way an assessment misses the thing it existed to catch.
- Errors are not symmetric in cost. A false positive on a fraud model inconveniences a customer; a false negative loses money. On a medical triage model the asymmetry runs the other way and is far more serious. Choosing a threshold is choosing which error to make more often, and that is a policy decision that should be made explicitly by someone accountable, not implicitly by whoever optimised the metric.
- Measure the harm, not the proxy for it. If the concern is that people are wrongly denied something, count the denials and follow up on them — do not infer it from a fairness statistic and stop there.
- Count the appeals and their outcomes. A process with no appeals is not working correctly; it is a process people cannot find or do not believe in. And a high overturn rate on appeal is direct evidence about the original decisions.
The fairness definitions genuinely conflict, and that is a theorem
You cannot satisfy every reasonable fairness criterion at once. This is proven, not a matter of engineering effort.
Equal accuracy across groups, equal false-positive rates, and a score that means the same thing across groups are mutually incompatible whenever the underlying base rates differ between those groups — which they usually do.
So ‘make it fair’ is not an implementable instruction. Somebody has to choose which definition applies, and that choice is a value judgement about which kind of error is worse for whom.
The governance move is to make the choice explicit and attributable — documented, with a name against it and a reason. A team that has quietly optimised one definition has made a policy decision without anyone noticing it was one.
And it is a genuinely strong interview answer, because it demonstrates that you understand the constraint rather than promising to engineer around it. Promising fairness in general is the answer of someone who has not read the literature.
Check yourself