AI from First Principles · Chapter 5 of 10
RAG — what it fixes and what it does not
The most common enterprise AI architecture, and the one whose risks are most often misplaced.
Retrieval-augmented generation is the standard answer to “the model doesn't know our stuff.” The mechanism is unglamorous: when a question arrives, search your documents, paste the best matches into the request alongside the question, ask the model to answer using them.
RAG is search plus paste. Understanding it that plainly tells you immediately where the risk lives — in the search index and the paste, not in the model.
What it genuinely fixes
- Currency — the model can use documents written after it was trained.
- Provenance — you can show which document an answer came from, making it checkable.
- Governance — data stays in a store you own, with your access control and retention.
What it does not fix
- It does not stop fabrication. The model can ignore retrieved text, blend it with recollection, or extend beyond it. Retrieval makes grounding possible, not automatic.
- It does not fix a bad index. Retrieve the wrong three documents and you get a fluent, well-cited, wrong answer — worse than an obviously wrong one, because the citation buys unearned trust.
- It does not inherit your permissions for free. Somebody implements that, per query, per user.
- It does not sanitise what it retrieves. Retrieved text enters the same channel as your instructions — which is chapter 8.
The five audit questions for any RAG system
1. What is in the index, and does its classification match the source? 2. Is retrieval filtered by the asking user's permissions, filtered afterwards, or not at all? 3. When a source document is deleted or reclassified, what re-indexes it and how fast? 4. Are the citations shown actually the passages used, or reconstructed afterwards? 5. Can you reproduce a given answer — question, retrieved passages, output?
Question 5 separates a system you can audit from one you cannot. Ask it first.
Where retrieval systems actually fail
Retrieval is presented as the fix for hallucination, and it is the right architecture. But the failures move rather than disappear, and they move somewhere less visible: a retrieval system that retrieves the wrong thing produces a confident answer grounded in an irrelevant document, which is harder to detect than an obvious fabrication.
| Failure | What the user sees | Where to look |
|---|---|---|
| Nothing relevant was retrieved | A confident answer built from whatever came back | Does the system have a floor — a relevance score below which it says 'I don't know'? Most do not |
| Chunking split the answer | Half-right answers; the definition and its exception land in different chunks | Chunk boundaries. Splitting on a fixed character count cuts tables, lists and clauses in half |
| The right document ranked fourth | A plausible answer from a related but wrong document | Retrieval quality in isolation — measure recall at k before you look at answer quality |
| Stale index | Yesterday's policy, stated as current | When was it last rebuilt, and what triggers a rebuild? |
| Permission leakage | A user gets content they may not see | Whether filtering happens before retrieval or after. After is a data-exposure incident waiting to happen |
| Conflicting sources | One answer, confidently, from two contradictory documents | Whether the system can say 'these disagree'. Almost none can |
The evaluation that most programmes skip
Evaluate retrieval separately from generation. Almost nobody does, and it is the single highest-value habit in this area.
If you only measure end-to-end answer quality, you cannot tell whether a bad answer came from bad retrieval or bad generation — and the fixes are completely different. One is chunking, indexing and ranking; the other is prompting and model choice.
The retrieval measurement is simple: take a set of real questions, record which document actually contains each answer, and measure how often it appears in the top k retrieved. That number alone diagnoses most systems.
The number is usually much worse than teams expect, and it is usually the actual problem — while the team is busy adjusting the prompt.
One more: measure how often the system correctly declines. A system that never says ‘the documents do not cover this’ has no floor, and every out-of-scope question becomes a confident wrong answer.
On the permission point, because it is a genuine incident class rather than a quality issue: if documents are retrieved first and filtered afterwards, the model has already seen content the user is not entitled to, and anything it wrote from that content can leak through a summary even if the source is withheld. Filtering must happen inside the retrieval query, not after it. This is the first question to ask about any internal knowledge assistant.
Check yourself