If an agent solved the same problem fifty times, why is it still reasoning? When agent work should become code
The best autonomous agents should make parts of themselves unnecessary. Five questions decide when a repeated agent task should become a tested playbook, and when the reasoning is worth paying for.
Listen to this article · 5 min
AI-generated narration of the full article.
Two engineering accounts published on October 13 arrive at the same idea from opposite directions.
In Microsoft’s Azure Networking, an on-call agent investigates production incidents, and when a reasoning path succeeds often enough it becomes a deterministic playbook that runs with no model calls. “The agent becomes a discovery mechanism for new problems, not a permanent tax on old ones.” We analyzed that design in Microsoft’s network incident agent.
At Google, Gemini 4 Argon spent a night exploring a 205,000-line codebase and proposed six optimizations. Engineers kept two, and what shipped was ordinary code with tests: no model in the loop at run time. “The agent explores, the maintainer subtracts,” as the team put it in the account we covered in six proposals, two merged.
In both cases, the agent’s most durable output was not an answer. It was software that made the next answer unnecessary.
The thesis
Our view, and it is a view rather than a measured fact: a well-run autonomous agent should make parts of itself unnecessary over time. Agents are the right tool for problems that are new, ambiguous or too varied to specify. Once a problem stops being any of those, continuing to pay a model to reason through it is a cost and a risk that nobody chose. It is slower than code, more expensive per run, harder to test, and it changes behavior whenever the model is upgraded, the issue we raised in model retirement as the new API deprecation.
That does not make agent reasoning waste. The exploration is where the value was found. The mistake is never converting it.
Five questions
For any task an agent performs repeatedly, five questions decide whether it should stay with the agent or become a playbook, a rule, a smaller model or plain code.
| Question | If yes | If no |
|---|---|---|
| Has the same path succeeded many times? | Candidate for a playbook | Keep reasoning; it is still discovery |
| Are the inputs stable in shape? | Code can parse them | Keep a model step for interpretation only |
| Can the right outcome be checked mechanically? | Test it like software | Keep human review on the agent |
| Would an error be costly or hard to reverse? | Convert sooner; tested code is safer | Lower urgency |
| Does the environment change faster than code can? | Keep the agent; rules would rot | Convert |
The fourth question runs against intuition. Teams tend to keep the riskiest work with the most capable model. But when a risky action has a known correct path, a tested playbook with an approval gate is safer than a model that may take a different path next time. Autonomy for consequential actions is easier to justify when the action is deterministic, which is the logic behind earning autonomy per action class.
What conversion usually looks like
Conversion rarely removes the model entirely. The common result is what Microsoft’s engineer calls a “deterministic skeleton, probabilistic filling”: code decides and executes, and a model step does only the interpretation that code cannot, such as turning messy output into typed fields that a validation step then checks. Snowflake’s accounts payable design made a similar choice, keeping deterministic steps deterministic around the parts that need a model. Where a decision remains but generation is unnecessary, a calibrated decision model can replace a frontier model at a fraction of the cost.
Running it as a practice
Three habits make this routine rather than occasional:
- Log reasoning paths, not just outcomes. You cannot convert a pattern you cannot see. Record which tools an agent called, in what order, with what result, so repeated paths can be found.
- Review the agent’s history like a backlog. Once a month, list the most frequent successful paths and run them through the five questions. Each conversion is a change, authored from the agent’s history and approved by an engineer, with tests and rollback.
- Send failures back. When a playbook meets a case it does not cover, route it to the agent as a new problem. The loop between discovery and routine is the system.
A useful metric falls out of this: the share of last quarter’s agent work that now runs as software. If it is zero, the agent is not learning anything the organization keeps. If it rises steadily, the agent is doing its most valuable job, which is finding the work that no longer needs it.