AI from First Principles · Chapter 5
RAG — what it fixes and what it does not
The most common enterprise AI architecture, and the one whose risks are most often misplaced.
Retrieval-augmented generation is the standard answer to “the model doesn't know our stuff.” The mechanism is unglamorous: when a question arrives, search your documents, paste the best matches into the request alongside the question, ask the model to answer using them.
RAG is search plus paste. Understanding it that plainly tells you immediately where the risk lives — in the search index and the paste, not in the model.
What it genuinely fixes
- Currency — the model can use documents written after it was trained.
- Provenance — you can show which document an answer came from, making it checkable.
- Governance — data stays in a store you own, with your access control and retention.
What it does not fix
- It does not stop fabrication. The model can ignore retrieved text, blend it with recollection, or extend beyond it. Retrieval makes grounding possible, not automatic.
- It does not fix a bad index. Retrieve the wrong three documents and you get a fluent, well-cited, wrong answer — worse than an obviously wrong one, because the citation buys unearned trust.
- It does not inherit your permissions for free. Somebody implements that, per query, per user.
- It does not sanitise what it retrieves. Retrieved text enters the same channel as your instructions — which is chapter 8.
The five audit questions for any RAG system
1. What is in the index, and does its classification match the source? 2. Is retrieval filtered by the asking user's permissions, filtered afterwards, or not at all? 3. When a source document is deleted or reclassified, what re-indexes it and how fast? 4. Are the citations shown actually the passages used, or reconstructed afterwards? 5. Can you reproduce a given answer — question, retrieved passages, output?
Question 5 separates a system you can audit from one you cannot. Ask it first.