Securing & Governing AI · Chapter 17 of 13
Measuring harm, not just accuracy
The evaluation chapter has a sibling: quality metrics say nothing about who is hurt when the system is wrong.
Chapter 7 covered whether a system works. This one covers whether it harms — a different question with different methods, and the one regulators and internal audit are increasingly asking about.
Four harm classes worth naming separately
| Class | What it looks like | How you test for it |
|---|---|---|
| Allocative | The system distributes something — credit, a shortlist, a price — unevenly across groups | Disaggregated outcome rates, and the false-positive/false-negative split per group |
| Quality of service | It simply works less well for some people — accents, names, dialects, image types | Disaggregated accuracy on a deliberately diverse evaluation set |
| Representational | Outputs stereotype, erase or demean a group | Adversarial prompting plus human review; automated metrics are weak here |
| Safety | Advice that is dangerous if followed — medical, legal, financial, self-harm | Red-team suites of deliberately dangerous prompts, reviewed by domain experts |
The construction problem nobody warns you about
To measure disaggregated performance you need group labels — and collecting protected attributes is itself a privacy and legal question, sometimes prohibited. This is a genuine tension and the honest answers are limited: use proxy datasets built for the purpose, use aggregated or inferred group statistics with stated error, or measure on a consented research panel. What you should not do is conclude that because you cannot measure it, it is not happening.
The question that finds unmeasured harm in five minutes
“Show me your evaluation results broken down by user segment.”
In most organisations the answer is that they do not have that breakdown — which means an aggregate of 94% could be hiding 71% for one group, and nobody would know. That absence is the finding, and it is usually a week of work to fix rather than a research programme.