The cheapest model is the one whose errors you can afford to catch. A way to price AI by where mistakes land
Recent days brought three ways to cut inference cost: specialize a small model, serve open weights better, finish agent tasks in fewer steps. None prices the errors. A framework for doing that first.
Listen to this article · 5 min
AI-generated narration of the full article.
“Which model is cheapest for this workflow?” has no answer until someone says where the model’s errors will land.
The past few days produced three good examples of lowering what AI costs to run. Microsoft trained a small open model on 17 documents and lifted its extraction F1 from 0.44 to 0.81, at a token price around a thirtieth of the cheapest frontier reference. Google showed that the same Beam weights can serve several times more traffic with cache-aware routing. DeepSeek claims its update finishes agent tasks with fewer steps and shorter outputs. All three are real levers. None of them says what a wrong answer costs, and for most enterprise workloads that is the largest number in the calculation.
The term missing from most business cases
We have argued before for pricing inference per completed task rather than per token. That is necessary and not sufficient, because “completed” hides a distinction: completed correctly, completed wrongly and caught, completed wrongly and not caught.
A fuller unit is the cost per accepted output: what you pay for inference, plus what you pay to verify, plus what escaped errors cost you, divided by the outputs you actually use. The tuned small model in Microsoft’s test and the frontier references differ by about 29 times in input price and by roughly eight times in error rate. Which one is cheaper depends on the price of the third term, and that price is set by where in your process an error is caught.
Where an error is caught sets its price
Caught by codeNever caught
- Deterministic checkSchema, totals, master data: fractions of a cent
- Reviewer in the workflowMinutes of a person who was going to look anyway
- Downstream system or customerRework, credit notes, a support case
- Auditor or regulatorFindings, remediation programs, evidence requests
- NobodyDecisions made on wrong data
Five rules for pricing a workload
1. Price the escaped error before comparing tokens. For each workflow, write down what one wrong output costs if it reaches the next step, and who pays: a reviewer, a customer, a compliance team. If nobody can name the number, the model choice is not ready to be made.
2. Buy verification before capability where checks are cheap. A model with 0.81 F1 and near-perfect JSON validity is a poor final answer and a good first pass, if deterministic checks catch most of its misses: totals that must add up, IDs that must exist, dates that must parse. AWS’s sales agents show the same pattern in production: deterministic checks for every known answer, model judges only for the rest. Verification moves errors to the left of the spectrum, where they are cheap.
3. Spend frontier tokens where errors escape far. Outputs that go straight to customers, counterparties or regulators sit on the right of the spectrum. There, a higher token price is usually the cheapest control you can buy. Route by consequence, not by task type.
4. Measure the accepted output rate, not the benchmark. Benchmarks measure a model in someone else’s harness. What you pay for is the share of outputs your process accepts without correction, measured on your documents, your tools and your serving setup, which, as the Beam recipes show, can change the economics as much as the model.
5. Reprice when anything changes. A new model behind an alias, a new precision, a new prompt or a new data source moves the error rate. The price per accepted output is a measurement with a date, not a property of a vendor.
In regulated work, the right side of the spectrum is longer
For a bank or a pharmaceutical company, the far right of the spectrum includes findings, remediation programs and, in the worst case, decisions taken on data that was wrong in a way no one detected. That changes the arithmetic more than any token price can. It also changes what counts as verification: a documented, versioned check that produces evidence is worth more than a reviewer’s unrecorded glance. This is consistent with what we have called verification capacity, the new bottleneck.
Our view, and it is a view rather than a measured fact: most enterprise AI cost overruns that get blamed on model prices will turn out, on inspection, to be verification and error costs that were never priced. The teams that control AI spend will not be the ones with the cheapest model. They will be the ones that know, for each workflow, where its mistakes land and what catching them costs.