The context window is the model's entire working memory. Everything it knows about your request — instructions, retrieved documents, conversation history, tool definitions, tool results — has to fit in it.
Context engineering is deciding what goes in, in what order, and what gets dropped when it will not fit. It has a larger effect on quality than prompt wording, and it is where most production AI systems succeed or fail.
The budget
Diagram, as text
- 32k window leads to System prompt: never evicted
- 32k window leads to Tool definitions: select per turn
- 32k window leads to Retrieved chunks: best ranked last
- 32k window leads to History: recent verbatim, middle summarised
- 32k window leads to User message
- 32k window leads to Reserve for the response
Treat the window as a fixed budget with competing claims:
| Claim | Typical share | Notes |
|---|---|---|
| System prompt and instructions | Small, fixed | Stable, cacheable prefix |
| Tool definitions | Grows with tool count | Often larger than people realise |
| Retrieved context | Largest variable | Where compression pays |
| Conversation history | Grows without limit | The one that eventually breaks things |
| Reserved for output | Must be kept free | Forgetting this truncates answers |
That last row causes a specific, confusing bug: the request succeeds, the model begins answering, and the response stops mid-sentence because there was no room left to generate. Reserve output space explicitly rather than filling the window with input.
Long context did not kill RAG
Models now accept enormous contexts, and the recurring claim is that retrieval is therefore obsolete — put the whole corpus in the prompt.
It does not hold, for four reasons worth being precise about.
Cost scales with tokens. Sending 500,000 tokens on every request costs hundreds of times more than sending 2,000 relevant ones, for the same answer. This alone settles most cases.
Latency scales with prompt length. Prefill is compute-bound, so a very long prompt directly delays the first token.
Attention is not uniform. Models reliably use material at the beginning and end of a long context and are measurably weaker on the middle. Filling the window does not mean the model uses all of it.
Corpora exceed any window. Your document store is larger than any context, and growing.
The honest conclusion: long context changes the trade-off, it does not remove retrieval. What it buys you is slack — chunks can be larger, you can afford more of them, and multi-document reasoning becomes practical. Retrieval still decides which documents.
Ordering matters
Because attention is uneven, position is a design decision.
- Instructions first. They set the task and belong in the reliable region.
- Strongest retrieved passages at the edges — highest-ranked at the start and end, weaker material in the middle.
- The user's question last. Immediately before generation, in the other reliable region.
This costs nothing and measurably improves grounded answering.
Managing conversation history
The claim that grows without bound. Four strategies, and real systems combine them.
Truncation keeps the last N turns. Trivial, and it silently forgets what the user said at the start — including constraints they expect you to remember.
Summarisation compacts older turns into a running summary, preserving the substance of a long conversation in a fraction of the tokens. The cost is an extra model call, and detail is irretrievably lost.
Selective retrieval treats history as a searchable store: embed each turn and retrieve only the ones relevant to the current question. Powerful for long-running assistants, and it means an important detail from fifty turns ago can resurface.
Structured state is the one people underuse. Extract durable facts — the user's name, their plan, their stated preferences — into a small structured object carried in every prompt, separate from the transcript.
Facts (always included): { plan: "enterprise", region: "EU", timezone: "IST" }
Summary (older turns): "User is migrating from a competitor, concerned about downtime..."
Recent turns (verbatim): last 6 messagesThis layering is what makes long conversations feel coherent rather than amnesiac, and it is far cheaper than sending the full transcript.
Tool definitions are context too
Every tool you expose costs tokens on every request — its name, description and full parameter schema. Twenty tools with detailed schemas can consume a substantial share of the window before any content.
They also degrade selection accuracy: a model choosing among forty tools picks worse than one choosing among six.
The fix is to not send them all. Retrieve relevant tools per request, or group them and expose one group at a time based on the task. This is retrieval applied to capabilities rather than documents.
Failure modes to recognise
Context overflow. The request exceeds the window and errors, or silently truncates. Budget explicitly and count tokens before sending.
Lost in the middle. Information is present but unused because it sat in the weak region. Reorder.
Context poisoning. Something wrong entered the context early — a bad retrieval, a failed tool result — and every subsequent turn reasons from it. Long conversations amplify this, which is an argument for periodically rebuilding context rather than appending forever.
Distraction. Too much marginally-relevant material crowds out the passage that mattered. The fix is compression, not more retrieval.
Check your understanding
Common mistakes
Not reserving space for output. Answers truncate mid-sentence and the cause is unobvious.
Appending history forever. Works in testing, breaks in week two of real use.
Summarising facts that needed to be exact. Account numbers and constraints belong in structured state.
Sending every tool definition every time. Costs tokens and degrades tool choice.
Putting the most important passage in the middle. Free quality left on the table.
Assuming a bigger window removes the need to think. It raises the ceiling and changes nothing about relevance.
Practice
Design the context budget for a support agent with a 32,000-token window: a system prompt, tool definitions, retrieved documents, conversation history and the user's message. Allocate it and say what gives way under pressure.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Decide the eviction order before you allocate, because that is what the design has to survive.
Show a strong answer
Allocate, leaving genuine headroom:
- System prompt and safety instructions — 1,500 tokens, never evicted. These define what the agent may do; losing them is losing the guardrails.
- Tool definitions — 2,000 tokens. Easy to overlook as context, and they grow silently as tools are added. Ten tools with verbose schemas can exceed the system prompt.
- Retrieved documents — up to 12,000 tokens, typically 5–8 chunks after reranking.
- Conversation history — up to 8,000 tokens.
- User message — up to 2,000 tokens.
- Reserve for the response — 6,000 tokens. The most commonly forgotten allocation: the window is shared between input and output, and a prompt that fills it leaves no room to answer.
Eviction order under pressure, first to give way:
- Retrieved chunks, lowest-ranked first. Chunk 8 contributes little and is at the position models attend to least. This is the cheapest reduction.
- Conversation middle, replaced by a summary. Keep the last 3–4 turns verbatim.
- Tool definitions for tools irrelevant to this request. If the user is asking about billing, the calendar tool's schema is dead weight — select tools per turn rather than sending all of them.
- Nothing else. If it still does not fit, fail explicitly rather than silently truncating, because silent truncation removes something and nobody knows what.
The ordering within the prompt, which matters as much as the budget: system prompt first, then tools, then retrieved documents with the highest-ranked chunk last — closest to the question — then history, then the user's message. That places the strongest evidence at the end, where attention is reliable, and keeps the invariant prefix stable so it can be cached.
An agent works well in testing and degrades over long sessions. Describe the likely failure modes and how you would detect each.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Long sessions differ from short ones in what has accumulated.
Show a strong answer
Three mechanisms, all invisible in short tests:
- Context dilution. The window fills with tool outputs, retrieved chunks and prior turns, so the system prompt and the current question compete with thousands of tokens of history. Instructions that held at turn 3 are ignored at turn 30 — not because the model forgot them, but because they are one small part of a large input. Detection: log total context tokens per turn and correlate with quality. A quality curve that falls as token count rises is this.
- Error compounding. A wrong intermediate result enters the history and is treated as established fact by every subsequent turn. The agent then reasons confidently from a false premise, and each turn makes it more entrenched because it now appears repeatedly. Detection: sample long sessions and look for the first turn where an assertion is wrong, then check whether it is repeated afterwards. Its recurrence is the signature.
- Summarisation drift. If history is compressed, each summary is a summary of a summary. Details erode, and specifics — an order number, a stated constraint — quietly become vague. Detection: compare facts stated in early turns against the current summary and count what has been lost.
What I would add to make these observable: record the full context sent on every turn, not just the user message. Almost all long-session debugging is impossible without it, and it is the first thing teams omit because the volume is large. Sample it rather than skip it.
The structural mitigations: pin critical facts into a scratchpad that is never summarised away, cap history growth explicitly, and periodically re-assert the system prompt near the end of the context where it will actually be attended to.
Key takeaways
- The window is a budget with competing claims — and you must explicitly reserve room for the output.
- Long context did not replace RAG: cost, latency, uneven attention and corpus size all still favour retrieving.
- Retrieval is best understood as relevance filtering, which improves answers even when everything would fit.
- Order matters — instructions first, strongest passages at the edges, question last.
- Manage history with a layered approach: durable facts as structured state, older turns summarised, recent turns verbatim.
- Tool definitions consume context on every request and degrade selection; retrieve tools rather than sending all of them.
- Recognise overflow, lost-in-the-middle, poisoning and distraction — each has a different fix.
Further reading
- Nelson Liu et al., "Lost in the Middle: How Language Models Use Long Contexts" (TACL, 2024).
- Yao Fu et al., "Data Engineering for Scaling Language Models to 128K Context" (arXiv, 2024).
- Charles Packer et al., "MemGPT: Towards LLMs as Operating Systems" (arXiv, 2023) — context as a managed resource with paging.
- Machel Reid et al., "Gemini 1.5" technical report (2024) — long-context retrieval behaviour and its limits.
- Anthropic, "Prompt engineering: long context tips" (documentation) — ordering and placement effects.