Stop comparing token prices with GPU hours. Price inference per completed task, over the hours you pay for
A GPU that lost money over a week, an agent made 29x cheaper on the same model, a decision model that bills only input. Our framework for choosing how to buy inference.
Listen to this article · 5 min
AI-generated narration of the full article.
What does one finished piece of AI work cost your company?
Not one token, not one GPU hour: one task done correctly, such as a case triaged, a document reconciled or a form filled. Few enterprises can answer, because the numbers they compare are priced in different units. A hosted API quotes dollars per million tokens. A GPU quotes dollars per hour. A platform quotes per seat. Comparing them directly is how teams talk themselves into the wrong architecture, in both directions.
Three results published in the same week show why.
Three numbers from one week
A GPU that paid in the morning and lost over the week. A Microsoft engineer served an open-weight model on an Azure Spot H100. In the busiest three hours, it produced $24.35 of tokens for $4.65 of compute. Over the week, it produced $86 of tokens for about $98 of compute, because setup, loading, idle time and evictions are paid for too. Same hardware, same model, opposite conclusions, depending only on which hours you count.
A workflow that was 29 times too expensive on the same model. Asana’s browser agent cost at least $36.21 per run; fixing how it cached and trimmed its own history brought the same model to $1.24. A larger context budget also took the share of runs that finished from 3 in 18 to 18 in 18. A cheaper configuration that does not finish is not cheaper.
A decision priced apart from generation. Microsoft-Decision-1 charges $0.042 per million input tokens and nothing for output, because it scores options instead of writing text. Calls that are really decisions now have their own price curve.
Each result is narrow, and two are vendor-published. Together they say the same thing: the price on the rate card explains less of the bill than utilization, context design and completion rate do.
The unit: cost per completed task
Our proposal is a single unit for every option:
Total spend over a period, including idle capacity and the people who operate it, divided by the number of tasks completed correctly in that period.
Both halves matter. The numerator must include the hours you pay for and do not use, and the engineering time that keeps the system running. The denominator must count only tasks that finished and passed your acceptance check, not requests served. Under this unit, a self-hosted model at low utilization and a frontier API with a wasteful context policy can both look as bad as they are.
This extends a view we have held since we argued that AI is now a portfolio of models and that hours saved are capacity, not value: the measure has to follow the work, not the component.
Four ways to buy inference
| Buying mode | You pay for | Idle time costs you | Tends to win when |
|---|---|---|---|
| Managed API, per token | Tokens processed | Nothing | Traffic is bursty or unknown; the quality bar is high |
| Dedicated managed capacity | Reserved throughput | Everything you reserved | Volume is steady and predictable; you need an SLA |
| Self-hosted open weights | GPU hours, plus operations | Everything, including loading and evictions | Traffic is steady with long shared prefixes; custody requires it |
| Routed mix | Several of the above, plus the router | Depends on the mix | Tasks vary widely; many calls are small decisions |
The table is less a recommendation than a list of who carries the risk of an empty hour. With a per-token API, the provider does. With reserved or self-hosted capacity, you do, and the case for it rests on keeping that capacity busy.
Four measurements before you choose
Utilization by hour. Plot expected requests per hour across a week, not the daily average. A GPU needs a sustained floor of work; in the Azure experiment, it had to be busy about a quarter of the time just to break even at Spot prices.
Prefix share and cache hit rate. Agent traffic resends long histories. If most input repeats, cached input is cheap on every platform, and the option that caches best, not the cheapest per token, may win. In the Azure test, cached input made up 52% of the value served.
Completion rate per configuration. Measure the share of tasks that finish correctly for each model and context policy you are considering. Asana’s numbers show this can move by a factor of six on the same model.
Operating hours. Count the engineering time to patch, monitor, scale and restart a self-hosted stack, and the on-call burden. It is usually the largest line missing from self-hosting business cases.
With those four numbers, the comparison becomes arithmetic. Without them, it is a debate between a rate card and a GPU price, and the loser is whoever’s assumptions were never tested.
Our expectation, and it is a view rather than a fact: as decision models, prompt caching and routing mature, the per-token price of frontier models will matter less to enterprise AI budgets than the share of hours, tokens and runs that produce nothing. The teams that measure that share first will choose their infrastructure on evidence. The rest will choose it on the price sheet.