An agent's context needs a specification. Define four properties before you blame the model
This week's releases attacked agent context from four sides: business meaning, evidence quality, freshness and the ability to find nothing. Treat context as a product with owners and tests.
Listen to this article · 5 min
AI-generated narration of the full article.
When an enterprise agent gives a wrong answer, the review usually starts with the model. Was it the wrong one, was the prompt unclear, should we try the newer version? Often the model did exactly what it was asked with what it was given. What it was given was wrong: an outdated policy, a document the user should not see, a definition of “active customer” that finance stopped using last year, or five loosely related passages when the honest answer was “there is nothing on this.”
Three releases this week each attacked one of those failures, from different vendors:
- Meaning. NVIDIA’s Agent Knowledge Foundation includes an Auto-Ontology library that maps raw data to business concepts and their relationships, and Kumo models that predict from relational data.
- Evidence and abstention. Cohere’s Rerank 5 returns calibrated relevance scores, where 0.7 means a 70% chance a document is useful, and passage-level selection.
- Freshness. AWS halved the default refresh interval of OpenSearch Serverless search and vector collections to five seconds, so newly ingested data becomes searchable sooner.
None of them is the whole answer. Together they suggest a way to think about agent context that most programs lack: as a product with a specification.
Four properties, each with an owner and a test
| The question it answers | How to test it | Who owns it | |
|---|---|---|---|
| Meaning | Do our terms mean one thing, and is it the current one? | Definitions resolved against the approved glossary | The business owner of each definition |
| Evidence | Is what the agent read relevant, permitted and attributable? | Retrieval evals; permission checks per user | Content owners and the platform team |
| Freshness | How old can the context be when the agent uses it? | Time from a source change to being retrievable | Data engineering, per source |
| Abstention | Does the agent know when it found nothing usable? | Questions with no answer in the corpus | The product owner of the agent |
Meaning is the property most often missing. Agents read definitions literally. If “revenue” means one column to the sales dashboard and another to finance, the agent will pick one, confidently. We argued that business definitions are becoming shared agent infrastructure; an automatically generated ontology makes the question more urgent, because it produces definitions faster than anyone reviews them.
Evidence has three parts, and teams usually measure only the first: relevance, permission and attribution. The document has to answer the question, the user has to be allowed to see it, and the answer has to say where it came from. Relevance can be evaluated; we explained how when measuring retrieval as its own system. Permission has to be enforced per user, not per agent.
Freshness is a number, and it should be written down per source. AWS’s change from ten to five seconds is a default refresh setting, not an end-to-end guarantee: what matters is the time from a change in the source system, through ingestion and indexing, to the agent being able to retrieve it. For a pricing agent, minutes may be too slow. For a policy assistant, a day may be fine. Without a target, nobody notices when it slips.
Abstention is the property that turns retrieval from a ranking into a decision. If relevance scores mean probabilities, a threshold can mean “nothing here is good enough,” and the agent can say so, ask a clarifying question or route to a person. An agent that always answers from its top five results will always answer, including when it should not.
Why “specification” and not “best practices”
Because a specification can be tested and owned, and a best practice cannot. Each property above has a number or a check that can run every day, and a person whose job is to keep it true. When an answer is wrong, the review can ask which property failed before anyone touches the model. When a vendor changes something, such as the model behind a “stable” alias in Google’s Agent Search, the tests catch it.
It also reallocates the work. In many programs the AI team owns everything from the prompt down to the index, including definitions it has no authority over. A context specification gives meaning back to the business, freshness to data engineering and evidence to the content owners, and leaves the AI team responsible for what it can actually control.
Where to start
Pick one production agent and write its context specification on one page: the ten terms it depends on and who owns them, the sources it reads with their freshness targets, the evaluation set for retrieval with permission checks, and the questions it must refuse to answer. Run the tests weekly. The first run usually explains more wrong answers than a month of prompt changes.