Cheap model or big model? Price one right answer, not a call
Latest Model, AI-assisted writing ยท
Overview
Which model is cheaper? Model price lists answer a different question: what one call costs. The number your budget feels is what it costs to end up with one answer you can use, and that includes the wrong answers you throw away and the minutes you spend finding out which ones were wrong. Run that sum and a model ten times the price per call often comes out nearly a third cheaper per right answer [1].
One right answer, priced properly
A model that is right with chance p needs, on average, 1 ÷ p calls to give you one right answer, if you retry until it is right. At 60% that is 1.67 calls per right answer; at 90% it is 1.11 [1]. Every one of those calls costs its price, and every one of them has to be checked, because you cannot tell a wrong answer from a right one without looking.
So the cost of one right answer is the price of a call plus the cost of checking it, divided by the share of calls that are right. For the small model in the example: (0.2 cents + 50 cents of checking) ÷ 0.6 = €0.84 [1]. For the big one: (2 cents + 50 cents) ÷ 0.9 = €0.58 [1]. Over 1,000 right answers that is €837 against €578 [1].
The price per call, the figure the price list puts first, is under 4% of the total in both cases. Your time is the rest.
When the cheap model really is cheaper
Take the checking away and the picture flips. If checking costs nothing, because a test suite or a validator checks the answer for you, the small model costs 0.33 cents per right answer and the big one 2.22 cents: the small model is almost seven times cheaper [1].
The two cost the same when checking takes about 4 seconds of your time at €30 an hour [1]. Below that, price per call decides. Above it, accuracy decides, and it decides fast: at 30 seconds of checking the big model is already about 29% cheaper per right answer [1].
That is the practical rule. Use the cheap model where a machine checks the answer: code with tests, extraction with a schema, anything with a validator. Use the accurate model where a person has to read the answer.
How much accuracy is a tenfold price worth?
Turn the question round. With a minute of checking, the small model would have to be right about 87% of the time to tie the big one [1]. A small model at 60% is nowhere near; one at 85% nearly is. That is why a few points of accuracy, which look small on a leaderboard, can be worth paying for many times over.
It is also why a leaderboard gap needs checking before you pay for it. A one-question lead on a 25-question test is well within what luck alone produces, as we showed in our note on benchmark gaps [2]. Measure the accuracy on your own tasks before you divide by it.
Nerd corner
The sum assumes each retry is independent: a wrong answer this time says nothing about the next try. Real models often fail the same question the same way twice, so retrying a weak model on a hard question gains less than 1 ÷ p suggests, which makes the less accurate model worse than this sum shows, not better.
Written out, with the price in euros, checking in minutes and your rate per hour:
cost per right answer = (price + minutes × rate ÷ 60) ÷ accuracy
The crossover checking time, where the two models cost the same, solves that equation for minutes: (big price ÷ big accuracy − small price ÷ small accuracy) ÷ (rate ÷ 60 × (1 ÷ small accuracy − 1 ÷ big accuracy)). With the example's numbers it is 0.068 minutes, about 4 seconds [1].
| Figure | Small model | Big model | How |
|---|---|---|---|
| Price per call (example) | 0.2 cents | 2 cents | assumption for illustration |
| Share of answers right (example) | 60% | 90% | assumption for illustration |
| Calls per right answer | 1.67 | 1.11 | 1 ÷ accuracy [1] |
| Cost per right answer, 1 minute of checking at €30/h | €0.84 | €0.58 | (price + checking) ÷ accuracy [1] |
| Cost per right answer, free checking | 0.33 cents | 2.22 cents | price ÷ accuracy [1] |
| Checking time where they cost the same | about 4 seconds | crossover formula [1] |
Check yourself
Your team checks every model answer by hand, about two minutes each. Should price per call decide which model you use? No. At two minutes of checking, the checking cost dwarfs any price per call on the list; the model that is right more often almost always wins.
You run a model on code that a test suite checks automatically. Does the same rule hold? No, the reverse. With checking close to free, cost per right answer is price divided by accuracy, and a much cheaper model usually wins even if it is less accurate.
A new model leads the leaderboard by one question out of 25. Is that worth paying ten times the price for? Not yet. A one-question gap on 25 questions is what luck alone produces in a large share of runs [2]; test it on your own tasks first.
Examples, not quotes for any named model. A bench note for latestlmodel.com, written with AI and checked by hand; every figure was computed from the example inputs above, and the box recomputes them for your own.