What is context rot?
Context rot is the gradual decline in an AI model's output quality as its context window fills with more text. A model may advertise a window of hundreds of thousands of tokens, but its ability to use every part of that window reliably drops well before the window is full. The information is still technically in view; the model just gets worse at finding it, weighting it, and reasoning over it as the input grows.
This is different from running out of context. When you exceed the window, the oldest tokens fall out of view entirely. Context rot happens inside the limit: everything still fits, yet accuracy, instruction-following, and consistency degrade as the conversation or document piles up. A long, cluttered context can produce worse answers than a short, focused one that carries the same key facts.
In plain words
Picture a four-hour meeting. Everything anyone said is technically still "in the room," but by hour three people misremember the decision from hour one, blur it with a tangent from hour two, and latch onto whatever was said most recently. Nothing was forgotten on purpose. The sheer volume just made the important point harder to hold onto. Context rot is that effect for an AI model: the more you pile in, the shakier its grip on any single part of it.
Why context rot happens
- Attention gets diluted. Every extra token competes for the model's focus. As the input grows, the signal you actually care about becomes a smaller slice of the whole.
- Clutter crowds out the signal. Earlier turns, abandoned tangents, and raw tool output stay in the window and compete with the part that matters right now.
- The middle gets neglected. Models tend to attend most to the beginning and end of a long input and least to the middle, so facts buried in the center are the first to be overlooked.
- Contradictions accumulate. Over a long thread, instructions pile up and sometimes conflict. The model ends up averaging across them instead of following the latest one.
Why it matters
- Long chats drift. The longer a conversation runs, the more likely the model is to lose the original goal or contradict something it said earlier.
- More retrieval can backfire. In Retrieval-Augmented Generation, stuffing in more chunks often lowers answer quality rather than raising it. Precision beats volume.
- Agents are especially exposed. Agentic workflows accumulate tool results and step history across many turns. Left unmanaged, that context rots and the agent gets less reliable the longer it runs.
- You pay for the damage. You are billed per token, so a bloated context costs more and can make the answer worse at the same time.
How to reduce context rot
- Curate, do not dump. Feed the model the few facts it needs, not everything you have. A tight context beats a full one.
- Position matters. Keep the most important instructions and facts near the start or the end of the input, not buried in the middle.
- Compact long histories. Summarize earlier turns, and start a fresh thread when a conversation has drifted far from its purpose.
- Retrieve narrowly. Return fewer, better-matched chunks instead of everything that looks vaguely relevant.
- Restate the goal. Periodically repeat the current instruction so it stays fresh and outweighs stale context.
Related articles:
- What is a context window? - The limit context rot happens inside, before you ever run out of room.
- Context engineering: the new developer skill - The discipline of deciding what goes into the window, which is how you fight context rot.
- What is Retrieval-Augmented Generation? - Where over-retrieving is a common cause of rot.
- What are agentic workflows? - Multi-step agents accumulate context and are most exposed to rot.
- What is a token in AI? - The unit both the window and context rot are measured against.
Want to stay one step ahead?
Don't miss our best insights. No spam, just practical analyses, invitations to exclusive events, and podcast summaries delivered straight to your inbox.