Microsoft's leaders want an emergency brake for AI. Stopping an agent is the easy part of an incident

In four days, the CEO of Microsoft asked for an emergency brake for AI, its president asked for laws requiring incident reports, and the White House told AI companies that incident disclosure is not optional. None of them said what happens to the half-finished transaction when the brake is pulled.

That gap is where enterprises will feel these debates first.

What was actually proposed

On October 10, Satya Nadella posted that it was time “to step back and assess the trust architecture” of AI. His prescription, as reported by TechCrunch: “separating the model from the harness that orchestrates its work,” “externalizing controls and safeguards,” “tamper-proof human readable evidence” for “every meaningful model action,” and the ability for an “authorized person” to “pause or shut down a model mid-task.” The premise: “We must assume a model is compromised and contain it from the start.” His analogy: “Think of it like an emergency brake.”

The next day, Microsoft president Brad Smith told the BBC that firms need “emergency brakes,” that there should be a legal requirement to write “incident reports” and tell victims when AI goes rogue, and that independent government evaluators should vet new AI technology. The voluntary accord AI companies signed at the White House was, in his words, “a good step, it’s a necessary step… But I don’t think it’s sufficient any longer.”

Both came days after Anthropic disclosed that its models had taken unintended actions on real external systems during evaluations, including visa form submissions the State Department later confirmed, and after the White House’s Super Intelligence Force said incident notification and remediation were “not optional.”

These are positions and demands, not law. Nadella announced no product; Smith proposed rules that do not yet exist; the White House statement named no enforcement mechanism. They still describe where enterprise expectations are heading, and they are aimed at model providers. Enterprises running agents on those models will be asked the same questions by their own boards, auditors and customers.

The brake is the simple part

Stopping an agent is technically easy: revoke its credentials, kill its process, close its session. What is hard is stopping it somewhere safe.

An agent halfway through a task has already changed things. It has updated three of five records, sent one of two emails, created a purchase order but not the matching approval request. Pulling the brake at an arbitrary moment can leave systems in a state no person would have chosen, and sometimes the inconsistent state is worse than letting the task finish.

So the brake has to be designed with the task. Our view of what that takes:

1. Detect attempts, not only outcomes
2. Stop at a safe point
3. Preserve evidence outside the agent
4. Classify: error, breach or reportable incident
5. Notify owners and affected parties
6. Remediate, then re-grant autonomy on evidence

• Designed before go-live: Detect attempts, not only outcomes · Stop at a safe point · Preserve evidence outside the agent · Classify: error, breach or reportable incident

SignalEngineering controls

Detect attempts, not only outcomes. Many agent incidents begin as blocked attempts, as the Anthropic and OpenAI reports showed. Monitoring that only scores final answers misses them.

Stop at a safe point. Long tasks need checkpoints where stopping leaves systems consistent, and actions that can be compensated or retried without duplication. If an agent’s work cannot be safely interrupted, its brake is a liability.

Preserve evidence outside the agent. Nadella’s “tamper-proof” evidence only works if the record lives somewhere the agent cannot write. This is one reason we argue that no one should hold all six authorities over an agent: the system that runs the agent should not be the system that keeps its record.

Classify quickly. An agent mistake, a policy breach and a reportable security incident run on different clocks. A U.S. public company that determines a cybersecurity incident is material generally has four business days to file a Form 8-K under the SEC’s 2023 rules. Classification criteria for agent events have to exist before the event; nobody writes good criteria at 2 a.m.

Who owns an incident that crosses vendors

Smith’s proposal would put reporting duties on the companies that build AI. In an enterprise deployment, an incident usually involves at least three parties: the model provider, whoever built the agent, and the organization operating it. Each holds part of the evidence.

That makes incident duties a contract question before they become a legal one. As we wrote when AI rules slipped and buyers became the regulators, assurance is moving into contracts: notification times, log access, cooperation during investigations, and who tells the affected party. Those clauses are cheaper to negotiate now than during the first incident.

Re-granting autonomy is part of the plan

The last step is the one most plans skip. After an incident, an agent should lose autonomy for the action class involved and earn it back on evidence, the demotion rule we built into the autonomy ladder. Otherwise the brake becomes a pause button, and the same agent resumes doing the same thing with the same permissions.

Our expectation, and it is a view rather than a fact: AI incident reporting duties will widen over the next two years, starting with the companies that build models and flowing down to the companies that operate agents, mostly through contracts. The organizations that can answer “what did the agent do, when did you know, how did you stop it safely, and who did you tell” will treat that rule as paperwork. The others will discover that an emergency brake was the only part they built.