Lakshya

AI from First Principles · Chapter 6

Agents, tool use, and where risk concentrates

The highest-risk pattern in production AI — and where your existing instincts work best.

An agent is a model given tools — functions it can call — and a loop: look at the situation, choose a tool, see the result, choose again. The tools are ordinary code: search a database, send an email, open a ticket, call an API.

This is where AI stops being a text generator and becomes a process that acts on your systems with a credential. Everything you know about service accounts, least privilege and blast radius applies directly.

IDENTICAL ATTACK. THE ONLY DIFFERENCE IS WHAT THE TOOLS COULD DO. agent reads poisoned input and follows the injected instruction SCOPED TOOLS read-only, one mailboxno send, no deleteno network egressevery call logged and traced outcome: a wrong summary BROAD TOOLS service account, all mailboxessend + delete permittedopen outbound HTTPtool calls not logged outcome: exfiltration at scale The model is not the control surface. The credential it holds is. This is a permissions review, and you already know how to do one.
The same successful attack produces a wrong summary or a mass exfiltration, and the model is identical in both. The only variable is what the tools were permitted to do — which is why agent security is a permissions review before it is anything else.

The confused deputy, back again

The agent acts with its permissions, not those of whoever asked. If it holds a broad service account and any user — or any document it reads — can influence what it does, that party has effectively borrowed the service account. The mitigation is old and well understood: act with the requesting user's authority, scoped per request, not with a standing super-credential.

The five controls, in priority order

  • Scope every tool to the narrowest permission that works. Read-only by default. One mailbox, not all mailboxes. Worth more than everything below combined.
  • Human approval on anything irreversible. Sending, paying, deleting, publishing, granting access. Not everything — that destroys the value — just the one-way doors.
  • Log every tool call with arguments, result and identity. If you cannot reconstruct what the agent did, you cannot investigate or audit.
  • Bound the loop. Maximum steps, spend and time. An unbounded agent with a retry loop is a denial-of-service against your own budget.
  • Separate reading untrusted content from holding power. The agent that reads the inbox should not be the one with write credentials.
← RAG — what it fixes and what it does notEvaluation — how anyone knows it works →