Before an agent runs a system, it should be able to explain it. Most systems don't keep enough history
The most useful first agent in a complex system is often read-only: it explains what happened. That depends on four records most platforms keep badly, and finding the gaps is the first deliverable.
Listen to this article · 5 min
AI-generated narration of the full article.
Most agent programs start by asking what the agent should do. In complex systems, the better first question is what the agent can explain.
Bayer Crop Science gave a clean example last week. Its platform runs more than 500 machine learning model configurations for plant breeding, and the problem was not running them; it was that nobody but engineers could say why a result looked the way it did. The agents Bayer built are read-only and explain what the platform did. Bayer reports that about 80% of questions are now answered without an engineer.
The pattern is not unique. NVIDIA’s Minerva maps a GPU cluster and its jobs into one graph so an operator can trace a slow run to a component. Operations agents from AWS and Microsoft produce a diagnosis that is a starting point, not a finding. Microsoft is building semantic models in Dataverse so agents understand what records mean before they touch them. In each case, the first valuable thing the AI does is make an existing system legible.
Our view, stated as a view: in a complex operational system, the explanation agent should come before the acting agent, and the main thing it delivers in its first months is not answers. It is a map of what the system cannot explain about itself.
An explanation needs four records
When someone asks “why did this happen?”, an agent can only answer from what was recorded. Four records decide whether the answer is an explanation or a plausible story:
What ran. Run history with inputs, parameters, start and end times and outcomes, including failed and retried runs. Many platforms keep the last successful state and little else.
What was configured. Versioned configuration, so “what changed since last season” has an answer. Configuration edited in place, or held in someone’s notebook, makes the agent guess.
Where the data came from. Lineage from source to output. Without it, “was the input different?” turns into a search the agent cannot do reliably.
What it means and who owns it. Business definitions and an owner per component. An agent that can name the person responsible for a pipeline is more useful than one that summarizes its logs.
An explanation agent that lacks one of these does not fail loudly. It produces fluent answers with the missing piece filled in by inference. That is why the gaps have to be measured deliberately: track the questions the agent could not answer, or answered without a source, and sort them by which of the four records was missing.
Why this order lowers risk
A read-only agent avoids the questions that stall most agent projects: who approves an action, how to undo it, who is liable when it goes wrong. It can go into production sooner, with permissions that a security team can approve in one review.
It also builds the evidence for what comes next. We have argued that autonomy should be earned one action class at a time. An agent that has explained a thousand failures correctly, with sources, is the strongest case for letting it propose a fix. An agent that has never been asked to explain anything is being trusted on a demo.
And it forces the data repair that acting agents would need anyway. We noted that an agent’s first lasting deliverable is often the data it forces you to fix. An explanation agent finds those gaps without having acted on them first.
In regulated work, this is already the standard
Life sciences and banking teams will recognize the four records. Data integrity expectations in pharmaceutical manufacturing ask for records that are attributable, contemporaneous and original; model risk management in banking asks for documentation that lets someone else reproduce a result. An explanation agent is, in effect, a continuous test of whether a system meets that bar, run by a reader who asks more questions than any auditor.
That suggests a practical sequence for any complex system an organization wants to automate:
- Deploy a read-only explanation agent for the people who currently wait on engineers.
- Log every question it cannot answer from a recorded source, and classify the gap.
- Fix the records, starting with the gaps that block the most questions.
- Only then let an agent recommend actions, and later take them, one action class at a time.
The appeal of agents is that they act. The systems worth automating are the ones that can first explain themselves, and an agent is a good way to find out which ones those are.