Your AI gateway holds keys to every model and tool your agents use. Protect it like identity infrastructure
LiteLLM's compromise and eight fixes in the official MCP servers point one way: the parts between agents and systems now concentrate credentials. A framework for treating them as tier zero.
Listen to this article · 5 min
AI-generated narration of the full article.
Ask a security team to list the systems whose compromise would hand an attacker the rest of the company, and they will name the same few: the identity provider, the directory, the certificate authority, the privileged access tooling. Security architects call these tier-zero assets; Microsoft’s enterprise access model now calls the same layer the control plane. They get separate administrators, hardened admin workstations, the shortest patch windows and the most attention.
Our view, and it is a view rather than an established practice: in a growing number of companies, the AI gateway and the agent’s tool servers belong on that list, and almost nobody has put them there.
What these components now hold
Look at what the LiteLLM 1.107.0 malware went after, according to Microsoft’s analysis of the compromise: cloud credentials for three providers, Kubernetes and CI/CD secrets, SSH and signing keys, configuration for developer and AI tools including MCP integrations, and the gateway’s own database records, where the keys for every model provider live. Then look at how the attack started, according to the maintainers: tokens found in the logs of an internal gateway, where a coding tool’s responses had carried them.
A gateway is designed to concentrate. One place holds every provider key, routes every prompt, records every response, and increasingly enforces spend and policy. We argued that this makes it a policy enforcement point. The same concentration makes it a credential store with network access to every provider, and its logs a record of whatever passed through.
Tool servers concentrate in a different direction. A filesystem server holds a path into your files; a fetch server holds a route into your network; a database server holds a connection string. Eight fixes in the official MCP servers on October 25 were all about one thing: those servers reaching further than the access they were granted, including a test server that exposed environment variables to any web page.
Neither component looks like tier zero, because each was introduced as developer plumbing. That is precisely how they escape the controls tier-zero systems get.
Six controls, borrowed from how identity infrastructure is protected
| The control | What it prevents | |
|---|---|---|
| Inventory | Every gateway, MCP server and agent host, with the credentials each one holds and what they reach | Discovering the blast radius during the incident |
| Release provenance | Images pinned by digest, hash-locked dependencies, and checks on who triggered the build, not only which key signed it | A signed artifact from a hijacked pipeline |
| Short-lived credentials | Workload identity and expiring tokens instead of provider keys stored in a database | One stolen database becoming every provider's key |
| Logs as secrets | Redaction of tokens, restricted access and short retention for prompt and response logs | Logs becoming the attacker's starting point |
| Egress allow lists | Gateway, build and tool hosts reach named destinations only | Collected secrets leaving the network |
| Rotation drill | A tested runbook that rotates every key the gateway holds within hours | A recovery that takes weeks because nobody knows the dependencies |
Two of these deserve emphasis.
Release provenance has to include the requester. The poisoned LiteLLM artifacts were signed by the project’s own workflow. Signature verification answers “which key?”. The question that would have caught this is “which repository, which branch, which person triggered this release, and was that expected?”. Build provenance attestations can carry that information; most deployment policies do not check it.
The rotation drill is the honest test. Ask the team that runs your gateway how long it would take to rotate every model provider key, every virtual key issued to applications, the gateway’s database password and its cloud role. If the answer is “we would have to find out,” the gateway is already tier zero in consequence and not yet in practice.
Where this sits with regulators
For banks and other supervised companies, this is not a new obligation so much as an existing one reaching a new component. U.S. examiners already expect privileged infrastructure and third-party software to be inventoried and controlled; an AI gateway that holds keys to external providers is both. The point of naming it tier zero is to stop it from being reviewed as an application.
It also connects to how we think about the model supply chain: every route to a model passes through a component that holds the route’s credentials. The supplier map and the tier-zero list should share entries.
Questions for your next architecture review
- Which components hold credentials for more than one model provider or more than one internal system?
- Who can trigger a release of each of them, and would anyone notice an unexpected one?
- Do prompt and response logs pass through the same secret-handling rules as the credentials they may contain?
- Which MCP servers listen on a network port, and which run as local processes?
- When did you last rotate every key the gateway holds, and how long did it take?
We expect at least one more incident of this shape before the industry treats it as routine; that is a forecast, not a fact. The companies that will absorb it best are the ones that moved their AI plumbing onto the tier-zero list before it was needed.