Lakshya

AI from First Principles · Chapter 1

What a model actually is

Everything that follows depends on this one picture. It takes ten minutes and it is not difficult.

If you are going to sign off on AI systems, you need one accurate mental model of what is inside them. Not the mathematics — you will never need that — but the shape. Most bad AI security decisions trace back to a wrong shape: people picture a database that answers questions, then reason about it as if it were one.

your text "the cat sat on the" tokens numbers, not words the weights billions of fixed numbers frozen after training a probability for EVERY possible token mat 61%floor 22%roof 9%... 8% one is sampled "mat" — usually There is no database in this picture. The model does not store your documents, look anything up, or check whether the answer is true. It produces a plausible next token, then does it again. Everything else — retrieval, tools, memory — is machinery someone built around this loop. That is where your controls go. Security consequence: "the model leaked our data" almost always means a system around it did.
A language model is a probability machine, not a knowledge store. It converts text to numbers, multiplies them through a fixed set of weights, and produces a probability for every possible next token. One is sampled. Then it does it again. That is the whole loop.

Three consequences you can act on

  • The weights are frozen. After training, the model does not change because you talked to it. A conversation teaches it nothing. If someone says “it learns from our users”, they mean somebody is collecting those conversations and running a separate training job — a data flow you can find, review and gate.
  • It has no notion of true. No lookup, no citation, no verification. Plausibility is the only thing optimised, which is why it is confidently wrong rather than uncertain: confidence is not a signal it computes.
  • Everything interesting is around the model. Retrieval, memory, tools, filters — all ordinary software somebody wrote. That is good news: the controls you already know how to review are the controls that matter.

The sentence to carry into every review

“The model leaked our data” is almost always false as stated. A system around the model — a retrieval index with no access control, a log that captured prompts, a tool with a broad credential — leaked the data. Finding which one is a normal engineering investigation, and it is the investigation you should insist on.

What about images, audio and video

They differ in what a token represents, not in the shape. Input becomes numbers, numbers pass through frozen weights, output is sampled from a distribution. Wherever this book says “token”, read “the smallest unit the model works in” and the reasoning holds.

← All chaptersThe four things people call “training” →