AI from First Principles · Chapter 3
Why it sounds certain when it is wrong
Hallucination is not a bug awaiting a patch. Understanding why changes what you ask for.
People wait for the release that fixes hallucination. It is not coming in that form, because confident-and-wrong follows directly from the mechanism in chapter 1 rather than being a defect layered on top of it.
The model produces a probability distribution over next tokens and samples from it. A fluent, well-formed, entirely fabricated citation is high probability — citations look like that. A refusal or an expression of uncertainty is comparatively low probability unless training deliberately raised it. Nothing in the loop checks the world.
- Fluency is not evidence. The most confident-sounding output is often the most dangerous, because fluency is optimised and truth is not.
- Asking the model if it is sure is near worthless. The follow-up is generated by the same process. Self-reported confidence is another sampled output, not a measurement.
- The fix is architectural. Ground the answer in retrieved text you control, cite the source, make the citation checkable. That is chapter 5.
- Rates fall; they do not reach zero. Plan for a residual error rate as you would any effective-but-imperfect control — with detection and recovery, not an assumption of prevention.
The governance move this unlocks
Because the error rate is real and irreducible, the right question in a review is never “is it accurate?” It is “what happens on the occasions when it is wrong, and who finds out?”
That is answerable, testable and auditable. It also moves the conversation from a vague objection the business will overrule to a concrete control it can implement.