The Myth That a Bigger Context Window Means Perfect Memory

The number gets bigger every few months. A model that once read a few thousand words at a time now advertises a context window of a million tokens, sometimes two million, enough to swallow a stack of contracts or an entire codebase in one go. The pitch that comes with it is seductive: hand the model everything and it will remember all of it. That last part is the myth, and it is worth taking apart, because a lot of people are now trusting these systems with long documents on exactly that assumption.

What the research actually shows

The clearest finding here has a name that says it all: lost in the middle. In a widely cited 2023 study, Nelson Liu and colleagues fed models long collections of documents and moved the key fact to different positions. Recall traced a U-shape. When the important passage sat near the beginning or the very end, the model usually found it. When it sat in the middle, accuracy fell off a cliff, in some tests dropping below the level a model would reach with no long context at all.

If that pattern sounds familiar, it should. Psychologists have described the same shape in human memory for decades, the tendency to recall the first and last items on a list and blur everything in between. The machine is not remembering the way a filing cabinet remembers. Its attention is uneven, and the middle is where things slip.

Why the advertised number misleads

A context window is a capacity limit, not a promise of recall. A model can accept a million tokens and still handle a fact buried at the 50 percent mark worse than a smaller model handed just the relevant page. The window tells you what the system will read without erroring out. It does not tell you what it will reliably use.

None of this means long context is fake. The limitation has narrowed. Newer systems do better on needle-in-a-haystack tests, where a single planted sentence has to be retrieved from a mountain of filler, and some now find that needle wherever it hides. But retrieving one distinctive sentence is an easier task than reasoning over many facts scattered through a long document, and it is the second task that most real work depends on. Recent papers through 2026 keep finding that retrieval quality degrades as you approach the context limit, even in models that ace the simple version.

The practical lesson

Relevance beats volume. A model given the right ten pages will usually outperform the same model given the whole filing cabinet, because the ten pages leave nothing important stranded in the dead zone. This is why serious systems still bother to search, rank and trim their inputs rather than dumping everything into the prompt and hoping.

The honest way to read a giant context number is as a ceiling, not a guarantee. It is the same trap as assuming a bigger model is always a better one. More capacity helps only if the system can actually use it, and on long documents, the evidence says it uses the middle least of all.