Securing & Governing AI · Chapter 18 of 13
Human oversight, and what the law actually requires
The obligations that bite, separated from the ones people worry about.
Regulation of AI is often discussed as a wall. In practice, for most enterprise systems, the obligations are narrower and more concrete than the discourse suggests — and knowing which ones apply to you is the difference between a two-week exercise and a year-long programme.
Tiering is by use case, not by technology
The same model is unregulated in one application and high-risk in another. Under the EU AI Act the categories that matter are prohibited (a short list including social scoring and certain biometric uses), high-risk (employment, credit, education, essential services, critical infrastructure, law enforcement and similar), and everything else, which carries mainly transparency obligations. Most internal enterprise tooling — a summariser, a code assistant, a search interface — is not high-risk, and establishing that early prevents a great deal of unnecessary work.
What high-risk actually obliges you to do
- A risk management system across the lifecycle, not a one-off assessment.
- Data governance — documented provenance, representativeness, and known limitations of training and evaluation data.
- Technical documentation and logging sufficient to trace a decision, which is the evidence set from chapter 12 stated as a legal requirement.
- Human oversight designed so a person can actually intervene — and the override-rate test in chapter 15 is how you find out whether they can.
- Accuracy, robustness and cybersecurity appropriate to the purpose, declared.
The obligation people most often miss
Transparency to the person on the other end. Where someone is interacting with an AI system, they generally have to be told. This is cheap to implement and routinely forgotten in internal tools that later become customer-facing — and it is one of the easier things for a regulator or a journalist to check from outside.
The sequence that avoids wasted effort
Tier the use case first. An hour of analysis determines whether you owe a conformity assessment or a disclosure banner, and most organisations skip it and default to treating everything as high-risk.
Then build the inventory, because you cannot tier what you cannot list. Then map controls to whichever framework your customers or regulators require. Adopting a framework before the inventory exists is the most common way an AI governance programme consumes a year and produces a binder.