Google's new TPUs are built for confidentiality and memory, not just speed. Long-running agents set the brief
TPU 8tx adds hardware confidential compute for regulated customers, TPU 8i Max doubles memory bandwidth, and TPU Sync treats model state as infrastructure. What matters for enterprises.
Listen to this article · 5 min
AI-generated narration of the full article.
Chip announcements usually lead with speed. Google’s announcement on November 9 leads with two other things: who can see the data while it is being processed, and where a model keeps its memory. Both are consequences of how AI is now used in companies, with sensitive data and with agents that run for hours.
Google has now introduced four TPU systems in one year for the first time. TPU 8tx and TPU 8i Max are the new ones.
| Built for | Why an enterprise would care | |
|---|---|---|
| TPU 8t | Large-scale training | Mostly a concern for model builders |
| TPU 8i | Inference | Cost of serving models and agents |
| TPU 8tx | Training and tuning with hardware confidential compute | Sensitive data and model weights stay protected while in use |
| TPU 8i Max | Inference with long contexts | Twice the memory bandwidth for agents holding large state |
TPU 8tx: confidential computing reaches the accelerator
TPU 8tx uses the same silicon as TPU 8t, with custom AMD CPU hosts and hardware-level confidential computing, which protects data and model weights while they are in use, not only at rest or in transit. Google positions it for regulated enterprises and governments, and for small-scale training, fine-tuning, distillation and reinforcement learning on sensitive data without moving it between clusters. Google also claims up to 2x lower latency than TPU 8t.
This is the part that matters most for banks, insurers, health systems and public bodies. The common objection to tuning a model on confidential records in a public cloud has been that the data is exposed in memory during processing. Confidential compute on the accelerator addresses that objection technically. It does not settle the legal and contractual questions, but it changes the conversation with risk and compliance teams from “impossible” to “show us the attestation”. It is a direct answer to the question we called custody, the first question in enterprise AI architecture.
TPU 8i Max and the memory problem
TPU 8i Max puts two TPU 8i chips in one package, doubling compute, on-chip memory and memory bandwidth, and fitting twice the silicon into the same rack space. Google says it targets the memory bandwidth bottleneck of large context windows.
The same theme runs through the software. Agent Compute Engine suspends idle agent sessions and moves their working memory off the chip in milliseconds. TPU Sync, now open source, moves the KV cache, the model’s working memory for a conversation, across chip memory, host memory, local SSDs and shared storage as one hierarchy. In Google’s lab tests on its previous generation, it cut KV cache transfer latency by more than 90%, bringing a 235-billion-parameter model’s median transfer to about 230 milliseconds, and it plugs into the open-source serving engines vLLM, SGLang and llm-d.
For enterprises, the point is economic. An agent that holds a 500,000-token context for an afternoon, mostly waiting for tools and people, occupies expensive memory the whole time unless its state can be parked and restored. We noted the same lever when llm-d moved inference cost below the model.
The rest, briefly
Cloud Pathways, the distributed runtime DeepMind built to train and serve Gemini, is now generally available on GKE, and Google says it processed more than 130 million TPU chip-hours in the last 90 days. GKE dynamic slicing is generally available, with claims of up to five times faster job startup and up to 4.5 times faster recovery from failures. An open-source TPU agent suite in preview works with Claude Code, Cursor and Google Antigravity.
What to ask your cloud team
Most companies will not buy TPUs directly; they will meet these chips through managed services. Three questions still apply. Can our fine-tuning or distillation run under confidential computing, and what attestation will we get? How does the provider price idle agent sessions, and is their memory parked or held? And which of our agent workloads are limited by memory rather than compute? The answers will matter more for the bill than the headline speedups.