The next cut in AI inference cost may come from below the model. llm-d 1.0 shows where

An agent working on a code change may call the model forty times in one task. Each call carries almost the same context as the last one: the same repository files, the same instructions, the same tool definitions, plus a few new lines. If the serving layer sends call 23 to a different GPU than call 22, the work of reading that context is done again from scratch. Multiply by thousands of agents and the largest line on the inference bill is not the model. It is recomputation.

That is the economic argument behind llm-d 1.0, released on October 20. llm-d is an open source distributed inference project founded in 2025 by Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA, now governed by the Cloud Native Computing Foundation, with more than 500 contributors from over 25 organizations. It runs on Kubernetes above inference engines such as vLLM and SGLang.

What 1.0 adds

Earlier versions already provided prefix-cache-aware and load-aware routing, KV cache offloading, separation of the prefill and decode phases onto different replicas, and support for mixed hardware. Version 1.0 adds what a shared production fleet needs:

• Multi-tenant quality of service. Priority and fairness aware flow control, so latency-sensitive requests and bursty agent traffic can share one fleet. Mistral engineers describe using priority bands to protect paid traffic under load.
• Zero-downtime operations. A highly available router with hot standby, rolling updates, and prefill and decode replicas managed and upgraded as one unit.
• Agent-aware serving. Session-aware routing, scheduling that prioritizes turns within a conversation, and native support for Responses API state, so less is recomputed between an agent’s turns.
• A stable foundation: a v1 configuration API and a hardened request path, validated across clouds and accelerators.

1. Request arrives with tenant and priority
2. Flow control: which band, what share
3. Session-aware routing to the replica holding the context
4. Prefill: only new tokens if the cache hits
5. Decode and stream the answer

• Decided before the model does any work: Flow control: which band, what share · Session-aware routing to the replica holding the context · Prefill: only new tokens if the cache hits

Agent requestServing decisions llm-d makes

Reading the cost claims

IBM cites 5 to 10 times lower cost for context-heavy agentic workloads. The best documented example is a demonstration with GLM 5.2, an open-weight model of about 753 billion parameters, on widely deployed NVIDIA GPUs: more than 6.6 million output tokens per minute at peak, up to 3,000 concurrent coding agents without preemptions, and, at current cloud rental rates, roughly 5 to 10 times less per token than equivalent commercial API pricing, with the largest savings on input-heavy traffic. An IBM Research study reports an approximately 98% prefix-cache hit rate and a blended cost of about $0.11 per million tokens.

These are workload-specific results. A 98% cache hit rate is a property of a workload where most of each request repeats, which agent loops often are and many other workloads are not. The comparison is against API list prices, and it leaves out the engineers who run the fleet. A JetBrains quote that it runs llm-d in production behind its Junie Lite plan is the most useful evidence in the release, because it is a production claim rather than a benchmark.

What changes for an enterprise

The cost lever moves down the stack. For teams self-hosting open-weight models, as many began to consider when Reflection released Beam’s weights, serving architecture now shapes cost as much as model choice. The questions to put to a platform team are new ones: what is our prefix-cache hit rate per workload, and do agent sessions stick to the replicas holding their context?

Priority becomes a policy. Multi-tenant QoS means someone decides whose requests wait when the fleet is full. That is a business decision, close to the budget decisions agents already require, and it should have an owner rather than a default.

Upgrades become routine instead of incidents. Coordinated upgrades of prefill and decode replicas address a real operational risk: a fleet half on one engine version and half on another.

It is still infrastructure you run. llm-d is open source, not a managed service available everywhere. Google Cloud says it is integrating llm-d 1.0 into GKE, and CoreWeave customers can run it on its Kubernetes service, but a self-managed deployment needs Kubernetes and inference-serving skills that most enterprises do not have in depth. As with running open weights yourself, the savings are real only for organizations that can carry the operating duty. Faster serving also says nothing about whether the answers are correct; that measurement stays where it was.