The Myth That a Bigger Context Window Means a Better Memory

Every few weeks a lab announces a bigger context window, the amount of text a model can take in at once, and the number climbs another order of magnitude. The buzz around Google's forthcoming Gemini 3.5 Pro and its 2-million-token window is the latest example. The implied promise is seductive: feed the model an entire codebase, a year of email, a stack of contracts, and it will hold all of it in mind and reason over the lot. The claim worth examining is the one hiding underneath, that a larger window means a better memory.

It does not, at least not reliably, and the evidence on this is now hard to wave away.

What the research actually shows

The foundational finding has a name researchers use without irony: "lost in the middle." Models attend well to the start and end of their input and poorly to whatever sits in between. Accuracy on retrieval and question-answering tasks traces a U-shaped curve, strong at the edges and sagging in the centre, with drops of more than 30 percent when the relevant fact is buried mid-document.

This is not a quirk of one weak model. A 2026 study from Chroma, summarised in a widely shared writeup, tested 18 frontier systems, including ones advertising million-token windows, and found every single one degraded as the input grew. Accuracy fell by 30 to 50 percent well before the documented limit was reached. The researchers gave the effect its own label, "context rot," and noted a second trap: text that looks relevant but is not actively misleads the model, dragging performance down further than length alone would predict.

Why the window and the memory are not the same thing

A context window is a capacity, not a competence. It tells you how much text the model is allowed to read, not how well it weighs every part of that text when forming an answer. The two get conflated because the marketing speaks in tokens, a clean, growing number, while the harder truth lives in benchmarks like LooGLE v2, which test whether models can actually track long-range dependencies in real documents. They often cannot.

None of this makes long context useless. It makes it a tool with a grain. The practical answer the field has converged on is hybrid: rather than dumping a million tokens at the model and hoping, you retrieve the few hundred thousand that genuinely matter and let it reason over those. Engineers building serious systems treat the giant window as a ceiling to respect, not a feature to max out.

The takeaway

So when the next headline touts a window twice the size of the last, it is fair to be impressed and fair to stay skeptical. More room to read is real progress. It is not the same as a model that remembers, weighs and uses everything you give it. Until a benchmark shows the middle holding up as well as the edges, the bigger number is a promise about capacity, not about understanding.