AI from First Principles · Chapter 3 of 10
Why it sounds certain when it is wrong
Hallucination is not a bug awaiting a patch. Understanding why changes what you ask for.
People wait for the release that fixes hallucination. It is not coming in that form, because confident-and-wrong follows directly from the mechanism in chapter 1 rather than being a defect layered on top of it.
The model produces a probability distribution over next tokens and samples from it. A fluent, well-formed, entirely fabricated citation is high probability — citations look like that. A refusal or an expression of uncertainty is comparatively low probability unless training deliberately raised it. Nothing in the loop checks the world.
- Fluency is not evidence. The most confident-sounding output is often the most dangerous, because fluency is optimised and truth is not.
- Asking the model if it is sure is near worthless. The follow-up is generated by the same process. Self-reported confidence is another sampled output, not a measurement.
- The fix is architectural. Ground the answer in retrieved text you control, cite the source, make the citation checkable. That is chapter 5.
- Rates fall; they do not reach zero. Plan for a residual error rate as you would any effective-but-imperfect control — with detection and recovery, not an assumption of prevention.
The governance move this unlocks
Because the error rate is real and irreducible, the right question in a review is never “is it accurate?” It is “what happens on the occasions when it is wrong, and who finds out?”
That is answerable, testable and auditable. It also moves the conversation from a vague objection the business will overrule to a concrete control it can implement.
The four kinds of wrong, which need four different fixes
“Hallucination” is used for at least four distinct failures with different causes and different remedies. Treating them as one thing is why so many mitigation programmes disappoint.
| Kind | What it looks like | What actually fixes it |
|---|---|---|
| Fabrication | A confident, specific, invented fact — a case number, a citation, an API that does not exist | Retrieval. Put the real document in the prompt. Prompting for honesty barely moves it |
| Staleness | Accurate as of some earlier date, wrong now | Retrieval again, plus being explicit about the date in the prompt |
| Misreading the source | The document was present and the answer still misstates it | Better chunking and a smaller task. Long, badly split context is a common and under-diagnosed cause |
| Sycophancy | Agreeing with a wrong premise in your question | Fix the question. A leading prompt gets a confirming answer, and this is the failure most likely to be caused by the person complaining about it |
The fourth is worth dwelling on, because it is the one people do not see. If you ask ‘why is X the best approach?’, you will get reasons X is best — whether or not it is. The model is completing your framing. In an evaluation programme, leading prompts are a large and invisible source of apparently good results.
Why fluency and accuracy are not related
The single most important thing to understand: nothing in how a model produces text makes confident-sounding output more likely to be right.
Fluency comes from the same process regardless of whether the content is well-grounded. So the usual human signal — hesitation, hedging, vagueness as a marker of uncertainty — is absent. You lose the cue you have relied on your entire life for judging whether someone knows what they are talking about.
And the failure is worst precisely where you can least check it. On a topic you know well, you catch errors immediately. On a topic you consulted the model about because you do not know it, you cannot. The error rate is not higher there — your detection rate is lower, which produces the same outcome.
The operational consequence: the review effort has to be allocated by consequence, not by how wrong the output looks. Output that looks obviously wrong is already safe, because someone will catch it.
Check yourself