Securing & Governing AI · Chapter 16 of 13
The AI gateway — how enterprises actually serve models
The pattern most large organisations converge on, and the one almost no public writing describes.
Ask an enterprise engineer which model they use and you will often get a URL rather than a model name. That is not evasion — it reflects how these systems are actually built. Applications do not call OpenAI or Anthropic or a self-hosted model directly. They call an internal endpoint, and something in the middle decides what happens.
Why organisations converge on this
- Policy enforcement in one place. ‘No customer PII leaves our tenancy’ is a sentence in a policy document until something inspects the request. Forty teams implementing it independently produces forty interpretations and at least one mistake.
- Model portability. When a provider deprecates a version or prices change, you reroute at the gateway instead of asking forty teams to redeploy. This alone usually justifies it.
- Cost attribution. Without a gateway, cloud AI spend arrives as one line item nobody owns. With one, every call carries an application identity and the bill is allocable.
- A single audit trail. One place holds the prompt, the retrieved context, the model version, the response and the policy decisions — which is exactly the evidence set in Auditing an AI system.
- Quota and blast radius. One team's runaway retry loop cannot consume the organisation's capacity or budget.
What sits inside it
| Layer | What it does | What breaks without it |
|---|---|---|
| Authentication | Identifies the calling application, not just a shared key | Unattributable usage; no way to revoke one consumer |
| Quota & rate limit | Per-application budget and throughput ceilings | One retry loop consumes the shared capacity |
| Input filtering | PII and secret detection, prompt-injection heuristics, classification ceiling | Confidential data reaches a third party and you find out from the vendor's logs |
| Routing | Chooses a model by policy — sensitivity, cost, latency, region | Teams pick models by preference; data residency becomes unprovable |
| Output filtering | Harm categories, leakage of retrieved content, PII in responses | The model repeats something it should not have |
| Logging | Prompt, response, versions, decisions, identity, cost | Nothing is reconstructable and the system is unauditable |
| Caching | Deduplicates identical calls | Cost, and needless latency |
What to ask about one, in an interview or a review
- “Can an application bypass it?” This is the question. A gateway some teams route around is not a control — it is a suggestion. Egress policy should make direct calls to model providers impossible from the application network, and I would ask to see that rule rather than the intention.
- “What is logged, where does it live, and for how long?” The gateway log is now one of the most sensitive data stores in the organisation, because it contains every prompt anyone has ever sent. It routinely inherits none of the classification of the data inside it.
- “What happens when the filter is wrong?” Input filters produce false positives and there must be an exception path, or teams will build a workaround and the workaround becomes the bypass.
- “Who decides routing policy?” If a central team decides which model serves which use case, that team is making risk decisions on behalf of the business, and it should know it.
- “Does it add latency, and how much?” Filtering and logging on the hot path cost milliseconds. Teams route around slow controls, so the performance budget is a security property.
For candidates: this changes what your answers should sound like
If you are interviewing at a bank, an insurer, a healthcare organisation or a large GCC, assume this architecture. Answering “I would call the OpenAI API” signals you have only built at start-up scale. Answering “I would consume our gateway endpoint, and I would want to know what it filters, what it logs and whether I can be routed to a different model without notice” signals you have shipped inside a regulated enterprise.
It also reframes several other answers. Model updates arrive without your consent, so your evals must run against the gateway rather than against a pinned vendor version. Your cost story is a quota conversation. And your incident response begins with the gateway log, not with the application.