Three models, 25 tasks, one date they all got wrong
Latest Model, AI-assisted writing ยท
Our first model bench put three models through 25 tasks with checkable answers, every answer scored by a program, not by a person or another model [1]. GPT-4.1 got 96% right, GPT-4o mini 92% and GPT-4.1 mini 88%; GPT-4o mini did it at about a thirteenth of GPT-4.1's cost [1]. A headline would say GPT-4.1 wins. The numbers say something more useful: the lead is mostly noise, and the one question every model got wrong tells you more than the ranking does.
The scores, and what a one-question lead is worth
On 25 questions, one question is four percentage points [2]. GPT-4.1's lead over GPT-4o mini is exactly one question: 24 against 23 [1].
Suppose, as an example, that the two models were equally good, both right about 94% of the time. Run each on 25 questions and they would differ by at least one question in 75% of runs, and by two or more in 35% [2]. A one-question lead is what two equal models produce three times in four. It is not evidence that either is better.
The uncertainty around each score says the same thing. With 24 right out of 25, the 95% range for GPT-4.1's true accuracy runs from about 80% to 99%; for GPT-4o mini's 23 of 25, from about 75% to 98% [2]. The ranges overlap almost entirely.
To tell 96% from 92% with the usual confidence, the gap has to be about two standard errors wide [2]. At 25 tasks the gap is 0.6 standard errors [2]. It takes about 280 tasks per model to reach two [2]. That is the size our bench is growing toward; until then we report ties as ties.
What the scores do tell you: the price of a right answer
The cost difference is not noise. GPT-4o mini ran the same 25 tasks for about a thirteenth of GPT-4.1's price [1]. Divide each model's cost by its share of right answers and GPT-4o mini still comes out about 13 times cheaper per right answer, because the accuracy gap is too small to change that [2].
Whether that matters depends on who checks the answers. When a program checks them, as it did here, price per right answer decides, and the smaller model wins. When a person has to read every answer, a few points of accuracy can be worth paying for many times over, as we worked through in our note on cost per right answer [3].
The question all three got wrong
All three models answered the date question with 366 [1]. Not three different wrong answers: the same one.
The mistake is easy to see once you see it. The models seem to have matched "2024" and "leap year" to "366 days" without checking where the leap day falls. 29 February 2024 comes before 1 March 2024, so the twelve months that follow contain a 28-day February and have 365 days [2].
The shared wrong answer is the real finding of this run. It means the three models make the same kind of mistake on the same kind of question. Switching from one to another would not have caught it, and neither would asking the same model twice. A one-line date calculation in code would have.
Nerd corner
The chance of a gap between two equal models comes from the binomial distribution. If each model is right with chance p on each of n questions, the number right is binomial(n, p), and the chance that two independent runs differ by at least k questions is the sum, over every pair of scores at least k apart, of the product of their two probabilities. With n = 25 and p = 0.94 that is 0.753 for k = 1, 0.353 for k = 2 and 0.129 for k = 3 [2].
The ranges are Wilson intervals for 24/25 and 23/25 at 95% [2]. The tasks needed come from the standard error of a difference between two proportions, √((0.96 × 0.04 + 0.92 × 0.08) ÷ n), set to half of the four-point gap: n = 4 × 0.112 ÷ 0.04² = 280 [2].
| Figure | Value | Where it comes from |
|---|---|---|
| GPT-4.1, right answers | 24 of 25 (96%) | the bench run [1] |
| GPT-4o mini, right answers | 23 of 25 (92%) | the bench run [1] |
| GPT-4.1 mini, right answers | 22 of 25 (88%) | the bench run [1] |
| Chance two equal 94% models differ by 1+ question in 25 | 75% | binomial, our calculation [2] |
| 95% range for 24 of 25 | about 80% to 99% | Wilson interval [2] |
| Tasks per model to separate 96% from 92% | about 280 | standard error of a difference [2] |
| Days from 1 March 2024 to 1 March 2025 | 365 | the calendar |
Check yourself
Model A scores 18 of 20 and model B 17 of 20. Is A better? Not on this evidence. One question in 20 is a gap that two equally good models produce most of the time; you would need a few hundred questions to tell them apart.
Three models give the same wrong answer to one question. What does that suggest? That they share the mistake, so asking a second model is no check at all. A check that works differently, such as code for dates and arithmetic or a lookup for facts, is the one to add.
A cheaper model is 4 points behind on a benchmark of 25. When should you still choose it? When a program checks its answers, so wrong ones cost little; then price per right answer decides, and the cheaper model usually wins.
A bench note for latestlmodel.com, written with AI and checked by hand. The scores and costs are from our published run of 29 October 2026, with its questions, scoring code and answers; every other figure was computed from them, and the box recomputes them for your own numbers.
Sources
- Latest Model bench run of 29 October 2026: 25 checkable tasks, scored by code, with results.csv and methodology
- Our own calculation (2026-10-29) from the bench scores: binomial gap probabilities, Wilson intervals and sample size, each checked a second way (Monte Carlo, Clopper-Pearson, standard error recomputed at n)
- Cheap model or big model? Price one right answer, not a call