Context Windows and Tokens: Why AI Agents Forget
On this page
Learning Objective
Understand tokens and context windows well enough to predict when an agent will lose track of details, why long sessions degrade, what compaction does, and which habits keep sessions focused, cheap and accurate.
Concept Explanation
Language models read and write text as tokens, pieces of words that average roughly four characters of English or code. Everything the model considers at a given moment must fit in its context window, measured in tokens: the system instructions, your project instruction file, the conversation so far, every file the agent has read and every command output it has seen. Current frontier models offer windows of hundreds of thousands to a million tokens, which sounds enormous until an agent reads thirty large files and three long test logs.
Context fills faster than people expect. A single verbose test run can produce tens of thousands of tokens. A large generated file, a lockfile or a minified bundle can consume a significant share of the window in one read. Agents also accumulate the history of everything they tried, including dead ends. Because the model rereads the entire context on every step, a bloated session is slower and more expensive on every turn.
More importantly, quality degrades before the window is full. Models attend less reliably to details buried in the middle of long contexts, and early instructions compete with everything that came after. In practice you see this as an agent that forgets a constraint you gave at the start, repeats an approach it already tried, or starts editing files unrelated to the task. Long sessions that wander between several topics suffer most.
When a session approaches its limit, agents compact it: they summarize the conversation so far into a shorter note and continue from that. Compaction keeps the session alive but loses detail, and the summary decides what survives. You can also compact deliberately with a focus, or simply clear the session and start fresh, which is often the better choice when switching tasks.
The practical habits follow directly. Use one session per task and clear between unrelated tasks. Put durable knowledge in a project instruction file rather than repeating it in chat (Day 7). Point the agent at specific files instead of asking it to read whole directories. Keep noisy output out of the window by running tests with quiet flags or delegating exploration to subagents that return summaries (Day 18). And when a session has clearly gone off track, start again with a better prompt rather than piling corrections on top of a confused context.
| Context consumer | Typical size | How to control it |
|---|---|---|
| Project instruction file | 1k to 5k tokens | Keep it short, move details to on-demand files |
| Each source file read | 1k to 20k tokens | Point to specific files or line ranges |
| Test and build output | Up to tens of thousands | Quiet flags, show failures only, subagents |
| Conversation history and dead ends | Grows every turn | One task per session, clear between tasks |
| Tool definitions from MCP servers | 1k to 20k+ | Connect only the servers the task needs |
Worked Example: Estimating What Fits Before You Ask
This small script estimates the token footprint of files using the rough rule of four characters per token, so you can see why "read the whole src directory" is a bad instruction and which files are worth pointing to.
import pathlib, sys
CHARS_PER_TOKEN = 4 # rough average for English and code
WINDOW = 200_000 # pick your model's window; leave room for history and output
def estimate(root: str, pattern: str = "**/*.*"):
rows = []
for p in pathlib.Path(root).glob(pattern):
if p.is_file() and not any(part in {".git", "node_modules", "dist"} for part in p.parts):
tokens = p.stat().st_size // CHARS_PER_TOKEN
rows.append((tokens, str(p)))
rows.sort(reverse=True)
total = sum(t for t, _ in rows)
print(f"{len(rows)} files, about {total:,} tokens ({total / WINDOW:.0%} of a {WINDOW:,} window)")
for tokens, path in rows[:10]:
print(f"{tokens:>8,} {path}")
if __name__ == "__main__":
estimate(sys.argv[1] if len(sys.argv) > 1 else "src")Running it on a medium-sized project typically shows that the source tree alone exceeds the window several times over and that a handful of generated or vendored files dominate. The lesson for prompting is to name the files and functions that matter, and let the agent search for the rest.
Real-World Examples
- Chroma's 2025 "Context Rot" study measured 18 models and found that performance on simple retrieval and reasoning tasks declined as input length grew, even when the information needed was present and the window was far from full.
- Claude Code exposes commands to inspect context usage, compact the conversation with an optional focus, and clear it entirely, and its documentation recommends clearing between unrelated tasks to keep sessions accurate.
Review Questions & Answers
About four characters on average, so a 40 kilobyte source file is around 10,000 tokens.
Models attend less reliably to details as context grows, and early instructions compete with everything added afterwards. Compaction may also summarize the instruction away.
It summarizes the conversation so far into a shorter note so the session can continue within the window. The downside is that details not captured in the summary are lost.
One session per task with clearing between unrelated tasks, pointing the agent at specific files rather than whole directories, and keeping noisy output such as full test logs out of the main context, for example with quiet flags or subagents.