Frontier models now advertise context windows of one to ten million tokens. Marketing suggests you can drop an entire codebase, medical record, or corporate archive into a single prompt. Empirical results tell a subtler story.
The needle-in-a-haystack illusion
Needle-in-a-haystack tests, which insert a target fact into a long distractor context, are the canonical demonstration that long-context models work. They also flatter the models. Real questions rarely resemble a single unambiguous needle. They require synthesizing scattered evidence, resolving contradictions, and remembering constraints established thousands of tokens earlier.
On multi-hop retrieval tasks, accuracy drops sharply past a few hundred thousand tokens even for models that pass single-needle tests at ten times that length. The context window exists; the ability to reason across all of it does not.
Why attention thins out
Attention distributes softly over all tokens in the window. As the window grows, the average weight on any given token shrinks. Architectural tricks — sparse attention, ring attention, retrieval-augmented attention — mitigate but do not eliminate the effect. The result is that models are often confident and wrong when asked to combine information from distant parts of the input.
- Position bias: models overweight the beginning and end of the context and underweight the middle.
- Distraction: irrelevant content in the context measurably reduces accuracy on questions about the relevant content.
- Instruction drift: a system prompt at position zero is less influential at token 800,000 than at token 8,000.
When to use long context anyway
Long context is genuinely useful in three cases. First, when the underlying corpus is small enough that retrieval adds latency without accuracy gains. Second, when the task requires whole-document reasoning that fragments would destroy — long legal contracts, extended narratives, or code repositories where cross-file references matter. Third, as a fallback when your retrieval system misses relevant chunks.
For everything else, a well-tuned retrieval pipeline still outperforms a large-context prompt on both accuracy and cost. RAG did not become obsolete when Claude and Gemini shipped million-token windows; it became the workhorse layer underneath them.
“Long context is a capability. Long memory is a discipline. Confusing the two is one of the most expensive mistakes teams make.”
Practical guidance
Build evaluations that reflect your actual query patterns. If your workload is single-fact lookup, needle tests are a reasonable proxy. If it requires cross-referencing, multi-hop, or numeric aggregation, you must test on that shape of question directly. Do not trust the marketed window length as a memory guarantee.
Instrument production traffic. Log the position of retrieved evidence, the length of the effective context, and the error rate. In our experience, teams routinely discover that half of their long-context calls could be shorter without accuracy loss — and that a quarter of them are silently failing.
Key Topics
Extended Knowledge
- The lost-in-the-middle effect, first documented in 2023, still shows up in the latest frontier models with different severity.
- Ring attention and other distributed schemes make long context computationally feasible but do not solve the reasoning problem.
- Retrieval-augmented generation remains the most cost-effective way to combine large knowledge stores with strong reasoning.
Frequently Asked
Trust it to accept the input without erroring. Do not trust it to reason uniformly across the input without evaluation on your task.
No. RAG usually delivers better accuracy per dollar and lower latency. Long-context prompting complements RAG for whole-document tasks.
Run an evaluation where the target evidence is placed at varying positions in the context. If accuracy drops in the middle, you have a lost-in-the-middle problem regardless of the advertised window length.
