AI from First Principles · Chapter 6 of 10
Agents, tool use, and where risk concentrates
The highest-risk pattern in production AI — and where your existing instincts work best.
An agent is a model given tools — functions it can call — and a loop: look at the situation, choose a tool, see the result, choose again. The tools are ordinary code: search a database, send an email, open a ticket, call an API.
This is where AI stops being a text generator and becomes a process that acts on your systems with a credential. Everything you know about service accounts, least privilege and blast radius applies directly.
The confused deputy, back again
The agent acts with its permissions, not those of whoever asked. If it holds a broad service account and any user — or any document it reads — can influence what it does, that party has effectively borrowed the service account. The mitigation is old and well understood: act with the requesting user's authority, scoped per request, not with a standing super-credential.
The five controls, in priority order
- Scope every tool to the narrowest permission that works. Read-only by default. One mailbox, not all mailboxes. Worth more than everything below combined.
- Human approval on anything irreversible. Sending, paying, deleting, publishing, granting access. Not everything — that destroys the value — just the one-way doors.
- Log every tool call with arguments, result and identity. If you cannot reconstruct what the agent did, you cannot investigate or audit.
- Bound the loop. Maximum steps, spend and time. An unbounded agent with a retry loop is a denial-of-service against your own budget.
- Separate reading untrusted content from holding power. The agent that reads the inbox should not be the one with write credentials.
What changes when a model can act
A model that only produces text has a bounded failure: it can be wrong. A model that can call tools has an unbounded one: it can be wrong and then do something about it. Every governance question about agents follows from that single change.
- Errors compound instead of being visible. In a five-step task, a 90% per-step success rate is a 59% task success rate. Steps that look individually reliable produce a system that fails more often than not, and the arithmetic surprises people every time.
- The blast radius is whatever the credentials allow. An agent with a write-capable token can do anything that token can do. This is an access-control question long before it is an AI question.
- Prompt injection becomes a real attack rather than a curiosity. If the agent reads a web page, an email or a document, whatever is in that content reaches the model. Text-only, the risk is a bad answer. With tools, it is an instruction that gets executed.
- Attribution gets hard. When something goes wrong across twelve steps and four tools, working out which step caused it requires traces you had to build in advance.
- And retries are not free. An agent that retries a failed non-idempotent action has just performed it twice. Idempotency stops being a nicety.
The controls that actually matter, in order
1. Least privilege on the tools, not on the model. The question is never ‘is the model trustworthy’ — it is ‘what can this credential do, and would we be comfortable if it did all of it at once?’ Read-only by default; write access as a deliberate, separately-scoped decision.
2. A human gate on anything irreversible. Sending, paying, deleting, publishing, granting access. The gate is not because the model is unreliable; it is because the action cannot be undone, which would justify a gate for a human colleague too.
3. A step budget and a wall-clock timeout. Agents loop. Without a ceiling they loop expensively, and the first sign is the bill.
4. Traces you can read after the fact — every tool call, its inputs and its result. Without these you cannot answer ‘what happened’, which is the only question anyone asks after an incident.
5. Treat all retrieved content as untrusted input. A document the agent reads is data, never instruction. This is the same boundary that governs user input in any application, and it is routinely forgotten here because the content looks like prose.
Note what is not on this list: a better system prompt. Prompt-level controls are worth having and are not a security boundary — anything expressible in the prompt is also expressible in the content the agent reads.
The design consequence that follows from the compounding arithmetic: shorter agent loops beat longer ones, and a deterministic pipeline with a model at two well-chosen steps usually outperforms an open-ended agent given the same task. When a workflow is known, encode the workflow and use the model for the parts that genuinely vary. The impressive demo is the open-ended one; the system that works in production is usually the constrained one.
Check yourself