LLM Tokens and Context Windows Explained
On this page
Learning Objective
Explain what a token actually is, why the context window is a hard architectural boundary rather than a model spec sheet number, why attention cost grows quadratically with sequence length, and why a large advertised context window does not mean you can or should fill it.
Concept Explanation
A token is a subword unit produced by the tokenizer, not a word and not a character. Common English words are usually a single token, rarer words split into several, and whitespace, punctuation, and formatting all consume tokens of their own. The often-quoted heuristic of roughly four characters per token holds reasonably for ordinary English prose and falls apart everywhere else: code is token-dense because of punctuation and identifiers, JSON pays a token for every brace, quote, and colon, and non-Latin scripts frequently tokenize far worse, which is a real and under-discussed cost and context-length penalty for non-English workloads. The practical rule is to count tokens with the actual tokenizer for the model you are using rather than estimating from character counts, because the estimate is systematically wrong in exactly the cases that matter.
The context window is the maximum number of tokens the model can attend over in a single forward pass, and it covers everything: the system prompt, the conversation history, the retrieved documents, the tool definitions, and the generated output. That last inclusion trips people up constantly. A 128,000-token window with 127,000 tokens of input does not leave room for a long answer, and depending on the API you will either get a truncated response or an outright error. Treating the window as a budget with named line items - system, history, retrieval, output reserve - and enforcing that budget in code is one of the highest-value pieces of unglamorous engineering in any LLM system, because the alternative is an endpoint that works fine in testing and starts failing the day a user pastes a long document.
The reason long context is expensive rather than merely large is the attention mechanism itself. In standard self-attention, every token attends to every other token, so the computation and the memory for the attention matrix scale with the square of the sequence length. Doubling context length roughly quadruples that portion of the work. Modern serving stacks fight this hard with FlashAttention-style tiled kernels that avoid materializing the full matrix, with KV caching so previously generated tokens are not recomputed, and with architectural variants like grouped-query and multi-query attention that shrink the cache. Those techniques change the constant factors and the memory profile dramatically, but they do not make long context free, and the KV cache itself becomes a serious GPU memory consumer that directly limits how many concurrent requests a server can hold.
There is also a quality reason not to fill the window, independent of cost. The "Lost in the Middle" work from Stanford in 2023 showed that model performance on retrieving a relevant fact from a long context is strongly position-dependent: accuracy is highest when the relevant information sits at the very beginning or the very end of the context and measurably degrades when it sits in the middle, producing a U-shaped curve. Practitioners have reproduced the same shape repeatedly with needle-in-a-haystack style evaluations. The systems consequence is direct: stuffing fifty retrieved chunks into a prompt and hoping the model finds the right one is worse than retrieving five good chunks and ordering them deliberately, and it costs ten times more.
So the context window should be understood as three separate ceilings that happen to share a number. There is a hard ceiling, the API limit, beyond which the request simply fails. There is an economic ceiling, the point at which the per-request cost stops justifying the marginal quality, which is usually far below the hard ceiling. And there is an effective ceiling, the point beyond which additional context stops improving and often degrades answer quality because of position effects and distraction from irrelevant material. Good systems are designed against the effective ceiling and monitored against the hard one. This is also the honest answer to "why not just put everything in the context window instead of building retrieval": you can, it is often the right call for small corpora, and it stops being the right call well before the advertised limit.
| Context Budget Line Item | Typical Share | Who Controls It | Failure If Unbounded |
|---|---|---|---|
| System prompt and tool definitions | Fixed, 500 to 3,000 tokens | Your team, and it only ever grows | Silent per-request cost multiplier across all traffic |
| Conversation history | Grows every turn | The user, indirectly | Long sessions eventually exceed the window and hard-fail |
| Retrieved context | Often the largest share | Your retrieval config (chunk size x top-k) | Cost blowup plus lost-in-the-middle quality degradation |
| Output reserve | Must be subtracted up front | max_tokens setting | Truncated answers or an over-limit error at generation time |
Worked Example: Enforcing a Context Budget Instead of Hoping It Fits
The difference between a system that degrades gracefully under a long document and one that returns a 400 at the worst possible moment is usually a single budgeting function that runs before the model call, trims the lowest-value content first, and reserves output space explicitly.
def build_prompt(system, history, chunks, question, window=128_000, max_output=1_500):
# Reserve output space FIRST - it comes out of the same window.
budget = window - max_output - count_tokens(system) - count_tokens(question)
# Trim in priority order: history is the cheapest thing to lose.
history = trim_history(history, max_tokens=min(2_000, budget // 4))
budget -= count_tokens(history)
# Retrieved context fills what is left, best-scoring chunk first,
# and we stop at the EFFECTIVE ceiling, not the hard one.
effective_cap = min(budget, 12_000) # tuned from evals, not from the API limit
selected, used = [], 0
for chunk in chunks: # already ordered by rerank score
t = count_tokens(chunk.text)
if used + t > effective_cap:
break
selected.append(chunk)
used += t
# Position matters: put the strongest chunk last, adjacent to the question,
# to avoid burying it in the middle of the context.
selected = reorder_strongest_at_edges(selected)
emit_metric('prompt.tokens', count_tokens(system) + count_tokens(history) + used)
emit_metric('prompt.chunks_dropped', len(chunks) - len(selected))
return assemble(system, history, selected, question)Real-World Examples
- The "Lost in the Middle" study from Stanford researchers in 2023 measured multi-document question answering across varying context positions and found a consistent U-shaped accuracy curve, with models performing best when the relevant passage appeared at the start or end of the context and noticeably worse when it appeared in the middle, a result that held even for models explicitly marketed on their long-context capability.
- Google's Gemini 1.5 Pro shipped in 2024 with a context window of one million tokens, later extended in preview to two million, which is large enough to hold hours of video transcript or an entire codebase. It made the economic and effective ceilings the binding constraints rather than the hard one, since filling a million-token window on every request is priced and latency-shaped in a way that rules it out for most interactive products regardless of whether the model can technically handle it.
Review Questions & Answers
Tokenizers produce subword units whose density depends heavily on the content. The common four-characters-per-token heuristic is calibrated on ordinary English prose, so it under-counts badly for code and structured formats, where punctuation, braces, quotes, and identifiers each consume tokens, and for non-Latin scripts, which frequently tokenize into far more tokens per visible character. Since those are exactly the inputs most likely to be long, the estimate is wrong precisely where an accurate count matters most, which is why the real tokenizer should be used for budgeting.
The context window covers input and output together, so tokens spent on input are not available for generation. If the prompt is assembled to fill the window, there is no room left for the answer, and depending on the API the result is either a truncated response or a request that is rejected outright at generation time. Subtracting max_tokens from the budget before deciding how much history and retrieved context to include converts that failure into a graceful reduction in how much context is supplied.
In standard self-attention every token attends to every other token, so the attention computation and the attention matrix both scale with the square of the sequence length, making a doubling roughly a quadrupling of that portion of the work. Serving stacks mitigate this with tiled attention kernels such as FlashAttention that never materialize the full matrix, with KV caching so earlier tokens are not recomputed on each generated token, and with grouped-query or multi-query attention that shrinks the cache. These improve constant factors and memory usage substantially but do not remove the underlying scaling, and the KV cache itself becomes a hard limit on concurrency.
Three separate ceilings apply. The hard ceiling is the API limit. The economic ceiling is where per-request token cost and generation latency stop being justified by the marginal quality gain, and it sits far below the hard limit for anything interactive. The effective ceiling is where additional context stops helping because of position sensitivity and distraction from irrelevant material, so a prompt with the right five passages typically beats one with five hundred passages that include them. Retrieval exists to find the small amount of context that actually matters, which is cheaper, faster, and usually more accurate than relying on the model to do that search inside its own attention.