Securing & Governing AI · Chapter 20 of 13
Human oversight, and what the law actually requires
The obligations that bite, separated from the ones people worry about.
Regulation of AI is often discussed as a wall. In practice, for most enterprise systems, the obligations are narrower and more concrete than the discourse suggests — and knowing which ones apply to you is the difference between a two-week exercise and a year-long programme.
Tiering is by use case, not by technology
The same model is unregulated in one application and high-risk in another. Under the EU AI Act the categories that matter are prohibited (a short list including social scoring and certain biometric uses), high-risk (employment, credit, education, essential services, critical infrastructure, law enforcement and similar), and everything else, which carries mainly transparency obligations. Most internal enterprise tooling — a summariser, a code assistant, a search interface — is not high-risk, and establishing that early prevents a great deal of unnecessary work.
What high-risk actually obliges you to do
- A risk management system across the lifecycle, not a one-off assessment.
- Data governance — documented provenance, representativeness, and known limitations of training and evaluation data.
- Technical documentation and logging sufficient to trace a decision, which is the evidence set from chapter 12 stated as a legal requirement.
- Human oversight designed so a person can actually intervene — and the override-rate test in chapter 15 is how you find out whether they can.
- Accuracy, robustness and cybersecurity appropriate to the purpose, declared.
The obligation people most often miss
Transparency to the person on the other end. Where someone is interacting with an AI system, they generally have to be told. This is cheap to implement and routinely forgotten in internal tools that later become customer-facing — and it is one of the easier things for a regulator or a journalist to check from outside.
The sequence that avoids wasted effort
Tier the use case first. An hour of analysis determines whether you owe a conformity assessment or a disclosure banner, and most organisations skip it and default to treating everything as high-risk.
Then build the inventory, because you cannot tier what you cannot list. Then map controls to whichever framework your customers or regulators require. Adopting a framework before the inventory exists is the most common way an AI governance programme consumes a year and produces a binder.
What “meaningful human oversight” has to mean operationally
Every framework requires a human in the loop for consequential decisions. Almost none define what makes the human meaningful, and the gap between the requirement and the reality is where most assurance programmes quietly fail.
| Requirement | What it means in practice | How it fails |
|---|---|---|
| The human can understand the recommendation | They can see what drove it, not just the output | A score with no explanation. The reviewer has nothing to reason about |
| The human has time | The queue and the target are set so that considering a case is possible | Throughput targets that make careful review impossible, set by someone who never reviewed a case |
| The human can disagree | Overriding is as easy as approving, and is not penalised | Approval is one click; override needs a written justification. That asymmetry is the policy, whatever the document says |
| Disagreement changes something | Overrides feed back into the system | Overrides recorded and never analysed — so the same error recurs forever |
| The human is accountable | A named person, not 'the team' | Diffuse responsibility, which means nobody looks closely |
The three measurements that settle it
The override rate. Near zero means the control is not operating — it is indistinguishable from no review at all while appearing on paper as a strong safeguard. This is the cheapest and most revealing number in AI assurance and it requires no new instrumentation.
Time per decision. If the median is eight seconds, nobody is reviewing anything. Compare it against how long a considered decision actually takes — which you can establish by asking someone to do ten properly and timing it.
Outcomes of overrides. When the human disagreed, were they right? If overrides are usually correct, the model needs work. If they are usually wrong, your reviewers need better information — and either finding is more actionable than an accuracy figure.
And one request that settles the question faster than any of them: “show me a case where the human disagreed and the outcome changed.” If none exists, the oversight is decorative — and you have discovered that before an incident rather than during one.
On the legal side, be precise and dated. Obligations differ by jurisdiction, by risk tier and by date, and several widely-quoted deadlines have moved. State when you last checked the position rather than asserting one flatly — confident wrong statements about regulatory timing are common in this area and they are expensive.
Check yourself