LLM Systems Engineering · Chapter 17 of 20
Context engineering
The window is a budget, not a container — what competes for it, the four things you can do about it, and why more context is reliably worse.
The floor between prompting and the harness. Prompting designs the words; context engineering designs what the model can see at the moment it answers — which, in any system with retrieval or tools, is a far larger and more consequential surface.
The reframing
The phrase to keep is that a context window is a budget, not a container. A container invites you to fill it. A budget forces the question the discipline is actually about: given a finite allocation, what earns its place this turn?
And the counterintuitive part, which is measured rather than argued
More context is reliably worse past a point, not merely more expensive.
The retrieval chapter has the numbers: a model advertising 128K frequently has an effective length around half of that; models that ace needle-in-a-haystack fall below 50% when the needle and question share no wording; and measured degradation with position and distractor density — context rot — takes one frontier model from over 99% at short context to around 70% by 32K.
So “we have a million-token window, just put everything in” is not a strategy. It is a way of paying more for a worse answer, and the failure is silent — there is no error, just a slow decline in quality as the window fills.
The four failure modes, with names
- Rot — quality degrades with length and with the density of irrelevant material, independent of whether the answer is present.
- Distraction — the right passage is in there, and it is competing with forty that are superficially similar. Retrieving twenty chunks instead of five is the commonest self-inflicted version.
- Poisoning — a wrong fact enters the window early, is re-read on every subsequent turn, and gets built upon rather than corrected. This one is specific to multi-turn work and it is badly under-taught: a hallucinated file path in turn two becomes an assumption the agent defends in turn nine.
- Clash — two retrieved documents contradict each other, or a retrieved document contradicts your system prompt. The model resolves it somehow, and you have no say in how. The 2019 policy and the 2024 policy both retrieved, both cited.
Notice that three of the four are about what is present rather than what is missing. Most people's intuition about context is ‘did it have enough’. The failures that actually dominate production are ‘did it have too much of the wrong thing’.
The four operations
Write is the one teams skip and it is the cheapest. A tool that returns ‘wrote 4,200 rows to /tmp/out.csv, columns are X, Y, Z’ costs a dozen tokens; returning the rows costs forty thousand, on every remaining turn. The agent can read the file if it needs to. Externalising state rather than carrying it is the single highest-return habit in agent work.
Select is retrieval, seen from the other side. Everything in the RAG chapters is a context-engineering technique — hybrid search and reranking are not really about search quality, they are about getting the right five passages in instead of the wrong twenty. It also covers tool search: loading only the tools relevant to this turn rather than all forty.
Compress is lossy and should be a deliberate decision. Providers now ship two distinct mechanisms and they are frequently confused: context editing clears old tool results or thinking blocks outright, while compaction summarises earlier history into a shorter form. Clearing is honest about what it destroys; summarising is not, which is why a summary of a summary is where long-running agents quietly develop amnesia.
Isolate is the strongest real argument for multi-agent, and it is worth separating from the general enthusiasm the agent chapter pushes back on: a sub-agent that reads two hundred pages and returns three sentences keeps two hundred pages out of the main window permanently.
Order, not just contents
Where something sits in the window matters as much as whether it is there.
- Stable first, volatile last. Caching is a prefix match, so a timestamp near the top invalidates everything after it. This is a cost decision and an ordering decision at the same time — see cost optimisation.
- Reference material before the question. Long document, then the ask.
- Put the constraint that must not be forgotten near the end. Material at the extremes of a long window is attended to more reliably than material buried in the middle. A rule stated once, forty thousand tokens ago, is competing with everything since.
- Re-state the critical constraint on long runs. Inelegant, and it works. Some agent harnesses re-inject the task definition every few turns for exactly this reason.
Context and memory are different things
| Context | Memory | |
|---|---|---|
| Lifetime | This request | Across sessions |
| Where it lives | The window | A file, a database, a store |
| Cost | Paid on every turn it is present | Paid only when retrieved into context |
| Fails by | Rot, distraction, poisoning | Staleness, and retrieving the wrong memory |
Memory is not a bigger context window — it is a store plus a retrieval policy, and the policy is the hard part. What is worth remembering, when to update it, when to forget, and how to avoid confidently recalling something that stopped being true. A memory system with no forgetting rule becomes a source of context poisoning that persists across sessions, which is materially worse than the within-session kind.
What to actually do
- Budget it explicitly. Write down roughly how many tokens each component may have — system, tools, retrieved material, history. If you have never done this, do it once; the allocation is almost always a surprise.
- Truncate at the boundary, not afterwards. Cap tool output where it is produced. One unbounded read poisons the rest of the run.
- Retrieve fewer, better passages. Reranking is a context intervention as much as a search one.
- Externalise anything large. Files, not payloads.
- Keep the cacheable prefix byte-stable, and put everything that varies after it.
- Have a plan for turn fifty, not just turn three. Most context designs are tested on short conversations and fail on long ones, which is where they are actually used.
Measuring it, which almost nobody does
This is the part that turns opinion into engineering, and it costs almost nothing.
- Log the token composition per request — how much was system, tools, retrieved material, history. You cannot manage an allocation you have never looked at.
- Track how it grows over a session. Plot tokens against turn number. If it climbs linearly to the cap, your compression is not working.
- Run your evals at realistic length. A suite that only tests short conversations will not catch rot — and rot is precisely what production hits. Add long-context cases to the eval suite deliberately.
- When quality drops late in a session, suspect the window before the model. It is the more common cause and the cheaper thing to check.
The one-sentence version: treat the window as a budget you allocate rather than a space you fill; remember that most context failures are too much of the wrong thing rather than too little of the right thing; and work through write, select, compress, isolate in that order, because the first two cost you nothing and the last two cost you information.
Check yourself