Is that benchmark lead real? How big a gap luck alone makes
Latest Model, AI-assisted writing ยท
Overview
A new model beats the old one by four points on a 100-question benchmark, and the launch post calls it a leap. Run that benchmark on two models that are exactly as good as each other and one of them leads by four points or more about half the time [1]. This is how to tell a lead from luck, with every number computed in the open.
What luck alone looks like
Take two models that are equally good: each answers any question correctly with the same chance, say 80%. Score them on 100 questions and note the gap. Do it a thousand times. Every gap you see is luck, because there is nothing else for it to be.
The gaps spread wide. At 100 questions the gap between two such models has a standard error of about 5.7 points, so luck alone puts them four or more points apart in roughly half of the runs, and 95% of the runs stay within about 11 points [1]. A lead has to clear that band before it says anything about the models.
Grading the 82 against 78
The same arithmetic grades any claim. With scores of 82% and 78% on 100 questions, the standard error of the gap is 5.65 points [1], computed from each score's own uncertainty [2]. The 95% range for the true lead runs from about −7 to +15 points [1]. It includes zero. The benchmark cannot tell these two models apart.
Give both the same 2,000 questions instead and the standard error falls to 1.26 points; the same 4-point lead is then more than three standard errors from zero, and very unlikely to be luck [1].
How many questions a lead needs
To show a true 4-point lead near 80% with the usual test (5% significance, 80% power) takes about 1,570 questions [1], from the standard sample-size formula for comparing two proportions [3]. A 1.5-point lead needs about 11,000. Most popular benchmarks have a few hundred to a few thousand questions, which is why small reported gains so often vanish on the next evaluation.
| Claim | Questions | Lead | 95% range for the true lead | Verdict |
|---|---|---|---|---|
| 82% vs 78% | 100 | +4 | −7.1 to +15.1 | Inside the noise [1] |
| 82% vs 78% | 2,000 | +4 | +1.5 to +6.5 | Clears it [1] |
| 81.5% vs 80% | 500 | +1.5 | −3.4 to +6.4 | Inside the noise [1] |
The standard error of a gap between scores a and b on n questions is √((a(1−a) + b(1−b)) / n), and the questions needed are about (1.96 + 0.84)² × (a(1−a) + b(1−b)) / d² for a lead d [3]. Real benchmarks give both models the same questions, and models tend to miss the same hard ones, so the true noise is somewhat smaller than this; a paired test such as McNemar's uses only the questions where the two disagree [4]. Treat these figures as the honest upper end of the noise. And if a lab tried twenty prompt variants and reported the best, the lead also carries the luck of the best of twenty.
Check yourself
Why does a benchmark with scores near 50% need more questions than one near 95%? Because a(1−a) is largest at 50%: each answer is most uncertain there, so each question tells you least about the gap [2].
A 1.5-point lead on 500 questions near 80%: real? Not shown: the 95% range runs from about −3.4 to +6.4 points, and showing a lead that small would take about 11,000 questions [1].
Models A and B are illustrations, not real products. This is a "we ran it so you don't have to" entry for latestlmodel.com, written with AI and checked by hand; every figure was computed, and the box above recomputes the ones that depend on your numbers.