Lakshya

LLM Systems Engineering · Chapter 17 of 20

Context engineering

The window is a budget, not a container — what competes for it, the four things you can do about it, and why more context is reliably worse.

10 min read2 diagramsAll 20 chapters

The floor between prompting and the harness. Prompting designs the words; context engineering designs what the model can see at the moment it answers — which, in any system with retrieval or tools, is a far larger and more consequential surface.

The reframing

A CONTEXT WINDOW IS A BUDGET, NOT A CONTAINER Everything below competes for the same space — and, more importantly, for the same attention. A typical agent turn, allocated: toolssystemskills / rulesretrieved docsconversation + tool resultsthe question Only the red slice is what the user actually asked. Everything to its left is scaffolding you chose to put there, and pay for, every turn. GROWS EVERY TURNHistory and tool results accumulate and are resent in full.So cost rises with the square of the conversation, and quality falls as it fills.CANNOT BE UNSAIDA wrong fact written into the window early is re-read on every later turn.And the model tends to build on it rather than contradict it. This is context poisoning.COMPETES WITH ITSELFYour instructions sit alongside 40,000 tokens of retrieved text.A rule stated once at the top is a small voice in a crowded room by turn twelve.IS NOT FREE AT EITHER ENDYou pay to put it in, and you pay again on every turn it survives.Which is why what you DROP is as much a design decision as what you add.
Look at the width of the red slice. In a mature agent, the user's actual question is a few per cent of what the model reads.

The phrase to keep is that a context window is a budget, not a container. A container invites you to fill it. A budget forces the question the discipline is actually about: given a finite allocation, what earns its place this turn?

And the counterintuitive part, which is measured rather than argued

More context is reliably worse past a point, not merely more expensive.

The retrieval chapter has the numbers: a model advertising 128K frequently has an effective length around half of that; models that ace needle-in-a-haystack fall below 50% when the needle and question share no wording; and measured degradation with position and distractor density — context rot — takes one frontier model from over 99% at short context to around 70% by 32K.

So “we have a million-token window, just put everything in” is not a strategy. It is a way of paying more for a worse answer, and the failure is silent — there is no error, just a slow decline in quality as the window fills.

The four failure modes, with names

  • Rot — quality degrades with length and with the density of irrelevant material, independent of whether the answer is present.
  • Distraction — the right passage is in there, and it is competing with forty that are superficially similar. Retrieving twenty chunks instead of five is the commonest self-inflicted version.
  • Poisoning — a wrong fact enters the window early, is re-read on every subsequent turn, and gets built upon rather than corrected. This one is specific to multi-turn work and it is badly under-taught: a hallucinated file path in turn two becomes an assumption the agent defends in turn nine.
  • Clash — two retrieved documents contradict each other, or a retrieved document contradicts your system prompt. The model resolves it somehow, and you have no say in how. The 2019 policy and the 2024 policy both retrieved, both cited.

Notice that three of the four are about what is present rather than what is missing. Most people's intuition about context is ‘did it have enough’. The failures that actually dominate production are ‘did it have too much of the wrong thing’.

The four operations

THERE ARE ONLY FOUR THINGS YOU CAN DO ABOUT IT WRITE — put it outsideSave state to a file, a scratchpad, a board.The window holds a reference, not the payload.“Wrote 4,200 rows to /tmp/out.csv” costs 12 tokens.The rows cost 40,000, every remaining turn.SELECT — bring in only what is neededRetrieval. Tool search. Loading one skill file.Choose per turn instead of carrying everything.This is what RAG is, seen from the context side.Precision beats volume — every time it is measured.COMPRESS — make it smallerSummarise old turns. Truncate tool output.Compaction; half-life policies on stale results.Lossy by definition. Decide WHAT you are willing to lose.A summary of a summary is where agents get amnesia.ISOLATE — use a separate windowA sub-agent reads a lot, returns a paragraph.The main window never sees the raw material.The strongest genuine argument for multi-agent.And the only one that is about context, not org charts. TRY THEM IN THAT ORDER. Write and select are free and lossless; compress and isolate are not. Most teams reach straight for compression — and a summary you did not need is information you destroyed for nothing. The same shape as the eval ladder and the cost ladder: exhaust the free operations before you trade anything away.
Everything you will ever do about context is one of these four. The order at the bottom is the practical content.

Write is the one teams skip and it is the cheapest. A tool that returns ‘wrote 4,200 rows to /tmp/out.csv, columns are X, Y, Z’ costs a dozen tokens; returning the rows costs forty thousand, on every remaining turn. The agent can read the file if it needs to. Externalising state rather than carrying it is the single highest-return habit in agent work.

Select is retrieval, seen from the other side. Everything in the RAG chapters is a context-engineering technique — hybrid search and reranking are not really about search quality, they are about getting the right five passages in instead of the wrong twenty. It also covers tool search: loading only the tools relevant to this turn rather than all forty.

Compress is lossy and should be a deliberate decision. Providers now ship two distinct mechanisms and they are frequently confused: context editing clears old tool results or thinking blocks outright, while compaction summarises earlier history into a shorter form. Clearing is honest about what it destroys; summarising is not, which is why a summary of a summary is where long-running agents quietly develop amnesia.

Isolate is the strongest real argument for multi-agent, and it is worth separating from the general enthusiasm the agent chapter pushes back on: a sub-agent that reads two hundred pages and returns three sentences keeps two hundred pages out of the main window permanently.

Order, not just contents

Where something sits in the window matters as much as whether it is there.

  • Stable first, volatile last. Caching is a prefix match, so a timestamp near the top invalidates everything after it. This is a cost decision and an ordering decision at the same time — see cost optimisation.
  • Reference material before the question. Long document, then the ask.
  • Put the constraint that must not be forgotten near the end. Material at the extremes of a long window is attended to more reliably than material buried in the middle. A rule stated once, forty thousand tokens ago, is competing with everything since.
  • Re-state the critical constraint on long runs. Inelegant, and it works. Some agent harnesses re-inject the task definition every few turns for exactly this reason.

Context and memory are different things

ContextMemory
LifetimeThis requestAcross sessions
Where it livesThe windowA file, a database, a store
CostPaid on every turn it is presentPaid only when retrieved into context
Fails byRot, distraction, poisoningStaleness, and retrieving the wrong memory

Memory is not a bigger context window — it is a store plus a retrieval policy, and the policy is the hard part. What is worth remembering, when to update it, when to forget, and how to avoid confidently recalling something that stopped being true. A memory system with no forgetting rule becomes a source of context poisoning that persists across sessions, which is materially worse than the within-session kind.

What to actually do

  • Budget it explicitly. Write down roughly how many tokens each component may have — system, tools, retrieved material, history. If you have never done this, do it once; the allocation is almost always a surprise.
  • Truncate at the boundary, not afterwards. Cap tool output where it is produced. One unbounded read poisons the rest of the run.
  • Retrieve fewer, better passages. Reranking is a context intervention as much as a search one.
  • Externalise anything large. Files, not payloads.
  • Keep the cacheable prefix byte-stable, and put everything that varies after it.
  • Have a plan for turn fifty, not just turn three. Most context designs are tested on short conversations and fail on long ones, which is where they are actually used.

Measuring it, which almost nobody does

This is the part that turns opinion into engineering, and it costs almost nothing.

  • Log the token composition per request — how much was system, tools, retrieved material, history. You cannot manage an allocation you have never looked at.
  • Track how it grows over a session. Plot tokens against turn number. If it climbs linearly to the cap, your compression is not working.
  • Run your evals at realistic length. A suite that only tests short conversations will not catch rot — and rot is precisely what production hits. Add long-context cases to the eval suite deliberately.
  • When quality drops late in a session, suspect the window before the model. It is the more common cause and the cheaper thing to check.

The one-sentence version: treat the window as a budget you allocate rather than a space you fill; remember that most context failures are too much of the wrong thing rather than too little of the right thing; and work through write, select, compress, isolate in that order, because the first two cost you nothing and the last two cost you information.

Check yourself

What does context engineering design?What the model can see at the moment it answers — system prompt, tools, retrieved material, history, tool results — not just the words of the request.
Why is a budget the better metaphor than a container?A container invites you to fill it. A budget forces the question the discipline is about: given a finite allocation, what earns its place this turn?
Is more context better?No, and it is measured. Effective length is often about half the advertised window, and one frontier model falls from over 99% to around 70% by 32K. The failure is silent.
What are the four failure modes?Rot (degrades with length and distractors), distraction (right passage competing with similar ones), poisoning (a wrong fact early gets built upon), and clash (contradictory sources).
What is the surprising thing about those four?Three are about what is present, not what is missing. Production failures are dominated by too much of the wrong thing, not too little of the right thing.
What are the four operations, in order?Write (externalise), select (retrieve only what is needed), compress (summarise or clear), isolate (a separate window). The first two are lossless and free; the last two cost information.
What is the difference between context editing and compaction?Editing clears old tool results or thinking outright; compaction summarises earlier history. Clearing is honest about what it destroys — summarising is where long-running agents develop amnesia.
How do context and memory differ?Context lives for one request and is paid for every turn; memory lives across sessions and is paid only when retrieved. Memory is a store plus a retrieval policy, and the policy is the hard part.
What should you measure?Token composition per request, growth across a session, and evals run at realistic length — a suite testing only short conversations cannot catch rot.
← Building an agent, start to finishHarness engineering →