AI from First Principles · Chapter 1 of 10
What a model actually is
Everything that follows depends on this one picture. It takes ten minutes and it is not difficult.
If you are going to sign off on AI systems, you need one accurate mental model of what is inside them. Not the mathematics — you will never need that — but the shape. Most bad AI security decisions trace back to a wrong shape: people picture a database that answers questions, then reason about it as if it were one.
Three consequences you can act on
- The weights are frozen. After training, the model does not change because you talked to it. A conversation teaches it nothing. If someone says “it learns from our users”, they mean somebody is collecting those conversations and running a separate training job — a data flow you can find, review and gate.
- It has no notion of true. No lookup, no citation, no verification. Plausibility is the only thing optimised, which is why it is confidently wrong rather than uncertain: confidence is not a signal it computes.
- Everything interesting is around the model. Retrieval, memory, tools, filters — all ordinary software somebody wrote. That is good news: the controls you already know how to review are the controls that matter.
The sentence to carry into every review
“The model leaked our data” is almost always false as stated. A system around the model — a retrieval index with no access control, a log that captured prompts, a tool with a broad credential — leaked the data. Finding which one is a normal engineering investigation, and it is the investigation you should insist on.
What about images, audio and video
They differ in what a token represents, not in the shape. Input becomes numbers, numbers pass through frozen weights, output is sampled from a distribution. Wherever this book says “token”, read “the smallest unit the model works in” and the reasoning holds.
Three consequences that follow from the shape
If a model is a fixed set of weights that turns text into a probability distribution over the next token, several things everyone finds surprising stop being surprising. Each of these is a question you will be asked by someone senior, and each has a short honest answer.
- “Why does it give different answers to the same question?” Because the output is a distribution, and something samples from it. Set that sampling to always take the most likely token and it becomes near-deterministic — at the cost of noticeably worse writing, because the most likely next word, chosen every time, produces flat and repetitive text. Variety and reproducibility are the same dial.
- “Why can't it tell me where it got that?” Because there is no ‘where’. Training did not store documents; it adjusted weights. Asking for a source is asking the model to generate a plausible-looking citation, which is exactly what it will do. This is why retrieval exists: it puts a real document in front of the model so there is something actual to cite.
- “Why does it not know what happened last week?” The weights were fixed at a point in time. Everything after that has to arrive in the prompt.
- “Why is it worse at arithmetic than a calculator written in 1974?” Because it is predicting the characters of an answer rather than computing one. This is why the fix is a tool call, not a better prompt.
- “Why can't we just ask it whether it is sure?” Because expressed confidence is generated text too. A model will produce ‘I am certain’ with the same machinery it uses to produce the claim. It is not reporting an internal state.
The wrong mental model, and what it costs
The damaging picture is “a database that answers questions”. Almost every bad AI decision traces back to it.
If you hold that picture, then a wrong answer looks like a lookup error — something that could be fixed by correcting the record. So people ask for the model to be ‘corrected’, expect a wrong answer not to recur, and are baffled when it does.
The accurate picture is a very well-read improviser who has no notes. They have read enormously and remember the shape of what they read, not the text. Ask about something well covered and you get something reliable. Ask about something obscure, and rather than saying ‘I don't know’, they produce the most plausible-sounding thing — confidently, in the right register, with the right structure.
That picture predicts the behaviour correctly, including the awkward parts: why it is fluent and wrong at the same time, why it invents citations, and why it is most dangerous exactly where you are least able to check it.
Check yourself