A $2-an-hour H100 earned four times its cost in a busy morning. Over the week, it lost money
A Microsoft engineer served Qwen3.8-27B on an Azure Spot H100. Busy hours paid four to one; the full week did not cover the compute. Utilization, not the GPU price, decided it.
Listen to this article · 5 min
AI-generated narration of the full article.
For three hours on the morning of October 9, a single H100 in Azure served 1,434 requests to Qwen3.8-27B: 168.7 million input tokens and 1.27 million output tokens. At a public list price for that model, those tokens were worth $24.35. The three hours of Spot VM time cost $4.65.
That is the result most self-hosting arguments stop at. Wuyi Weng, who ran the experiment and published it on Microsoft’s Apps on Azure blog on October 10, kept measuring for the rest of the week.
| Measure | Busiest 3 hours (Oct 9) | First week (Oct 3 to 10) |
|---|---|---|
| Hours the model server was up | 3.0 | 37.9 |
| Requests | 1,434 | 5,469 |
| Input tokens (share from prefix cache) | 168.7M (88%) | 564.2M (88%) |
| Output tokens | 1.27M | 6.11M |
| Worth at list price | $24.35 | $85.94 |
| Compute cost | $4.65 | About $98 for 46 VM hours, plus disks and NAT |
Busy hours returned $8.12 of tokens per hour the server was up, against roughly $2 of VM. Over the week, that fell to $2.27 per server hour, and the VMs actually ran about 46 hours, counting setup, model loading, benchmarks and early runs before the telemetry existed. The week cost about $98 in compute to produce $86 of tokens. The author’s own answer to the title: “it pays off when it’s busy.”
Where the value came from
A detail in the data overturns a common habit. In the busy window, cached input produced 52% of the token value, freshly computed input 35%, and output only 13%. Agent-style traffic resends long conversation histories, and vLLM’s prefix caching served 88% of input tokens from cache.
Throughput benchmarks usually report output tokens per second. For this workload, output was less than a fifth of the value. If your agents resend long contexts, prompt processing and cache behavior matter more than generation speed, and a benchmark that ignores them measures the wrong thing. It is the same mechanism that cut Asana’s browser agent costs, seen from the infrastructure side.
The hours you pay for but cannot sell
The week’s loss did not come from slow inference. It came from hours with no paying traffic: loading the 31 GB checkpoint, benchmarking, idle time and interruptions.
Spot capacity sells spare GPUs at a discount, with no SLA and 30 seconds’ notice of eviction. The experiment saw seven evictions in about 40 hours of running, and five attempts to create or restart a VM in one region that failed for lack of capacity. The author also notes that Azure’s eviction rate is per hour: at 5% an hour, an eight-hour run has roughly a one-in-three chance of being interrupted. The post’s advice is to read even the lowest eviction bucket as a reason to plan for evictions, not as a sign that they are rare.
At the prices in this test, the GPU needed to be busy roughly a quarter of the time to break even. Every hour it was evicted, loading or idle was an hour it could not earn.
This was an experiment, not a production design
The author is clear about the boundaries, and they matter. The gateway had no authentication of its own; an IP allow-list on the tunnel was the only access control, which suited personal tests and nothing more. Spot VMs carry no availability SLA. Regular capacity would remove evictions at a higher hourly price, which moves the break-even point. And “worth at list price” is an equivalence, not money saved.
None of this makes self-hosting a bad idea. It makes it a utilization question. The author’s conditions for when it makes sense are worth keeping: steady traffic whose prompts share long prefixes, a real need to control the exact checkpoint and serving configuration, and a workload that tolerates a restart, or a budget for regular capacity.
That is a narrower case than “open weights are cheaper,” and an honest one. As we noted when Reflection released Beam’s open weights, with open weights the operating duty moves to you. This experiment puts a price on part of that duty: the hours nobody is using what you run. Before comparing a GPU’s hourly rate with an API’s token price, measure how many hours a day your traffic would actually keep the GPU busy.