Do AI models use their whole context window? Not the middle
Latest Model, AI-assisted writing ยท
Overview
Context windows keep growing, and it is tempting to read that as "paste everything in and the model will use it". A careful test of how models actually use long inputs found a pattern that matters for anyone building on them: where a fact sits in the prompt changes whether the model finds it.
What the test did
Nelson Liu and colleagues gave language models a question plus a set of documents, only one of which contained the answer, and moved that document through every position [2]. They also tested simple key-value retrieval. Performance was highest when the relevant information was at the beginning or end of the input, and it degraded significantly in the middle, even for models built for long contexts [2].
The numbers
For GPT-3.5-Turbo with 20 documents [2]:
| Where the answer sat | Accuracy [2] |
|---|---|
| First of 20 | 75.8% |
| 10th, the middle | 53.8% |
| Last | 63.2% |
| No documents at all | 56.1% |
Moving the answer from first to the middle cost 22 percentage points, a 29% relative drop [1]. In the middle, the model did 2.3 points worse than with no documents to read at all [1] [2].
Nerd corner
The relative drop is 1 − 53.8 / 75.8 = 0.290, or (75.8 − 53.8) / 75.8 = 22.0 / 75.8, the same 29.0% [1]. The models tested date from 2023 [2]. Newer models may handle long contexts better, and some vendors test for exactly this. The shape of the finding is still the right thing to check before trusting a long prompt: put a known fact at different depths and see whether the answers change.
Check yourself
Where in a long prompt was information used best? At the beginning or the end [2].
How much did moving the answer from first to the middle cost? 22 percentage points, a 29% relative drop [1] [2].
Did models built for long contexts escape it? No: the degradation held for them too [2].
A notebook entry for latestlmodel.com, written with AI and checked by hand. The accuracies are quoted from the paper; the drops are computed from them.