AWS's sales agents were healthy and financially wrong. The fix was building evaluation into every release
AWS published how it evaluates the multi-agent system its field sellers use: nine stages from defining quality to production canaries. The useful part is where each check sits.
Listen to this article · 6 min
AI-generated narration of the full article.
An AWS agent was asked for a customer’s revenue over three months. The figure in the data was $410.08. The agent reported it as “$410.08 Million.”
Nothing alerted. The agent responded, latency was normal and every dashboard stayed green, because the dashboards measured whether the agent answered, not whether the answer was right.
That example opens an unusually candid engineering account AWS published on October 16 about the multi-agent system its Sales, Marketing and Global Services operations team built for field sellers. Sellers ask questions in plain English through Amazon Quick; sub-agents handle performance insights, customer insights and text-to-SQL over sales data, with answers scoped to each user’s territory and role. The post lists the other failures that started the work: Sales Revenue labeled as Sales Revenue excluding Adjustments, an Annual Recurring Revenue estimate that appeared nowhere in the data, and the agent’s own calculations presented as sourced facts.
The useful part is not that AWS evaluates its agents. It is where each check sits.
“Correct” is written down before anything is tested
The first of AWS’s nine stages defines quality: business outcomes mapped to metrics and thresholds, latency and cost budgets, and six named types of accuracy failure. The built-in evaluator templates in Amazon Bedrock AgentCore did not cover those failure types, so the team wrote custom evaluators for them.
The order matters. Many teams start with a generic LLM judge that scores helpfulness, then discover that the failures that cost money, such as a units error in revenue, are not what the judge was looking for. The failure taxonomy is the evaluation design; the tooling comes after.
Deterministic checks where an answer is known, judges where it is not
Deterministic checks verify facts with known answers: revenue, renewal dates, contract status, whether the right customer entity was resolved. An LLM judge scores what has no single right answer, such as relevance, faithfulness and the quality of the reasoning. AWS also runs third-party evaluators from DeepEval and AutoEval.
That split is the right one. A judge model asked whether “$410.08 Million” is faithful to a table may or may not notice the problem; a comparison with the warehouse will. As Anthropic’s research on rewarding AI auditors showed, a model grader is a component with its own error rate.
- Foundation: define quality and failure types
- Experiment: compare versions
- Merge: sentinel set on every staging deploy
- Gate: full golden set with frozen tool responses
- Canary: scheduled evaluation in production
- Adversary: guardrails on inputs and outputs
- Drift: sample live traffic, A/B before rollout
- Pulse: traces, alarms, dashboards
- Feedback: failures become test cases
- Before release: Foundation: define quality and failure types · Experiment: compare versions · Merge: sentinel set on every staging deploy · Gate: full golden set with frozen tool responses
- In production: Canary: scheduled evaluation in production · Adversary: guardrails on inputs and outputs · Drift: sample live traffic, A/B before rollout · Pulse: traces, alarms, dashboards
- Back to development: Feedback: failures become test cases
Where most teams under-investWhat catches failures nobody shipped
The 38% window
One finding deserves a place on every platform team’s wall. In a pre-production test, a data change, a tool change and a matching prompt update were meant to ship together and instead landed at different times. During the mismatch, the pass rate fell from 78% to 38%. No component was broken. The combination was.
An agent release is not a prompt. It is the prompt, the tool definitions, the tool code, the data the tools read and the model, and they have to be versioned and promoted as one unit. If your process lets the prompt team ship on Tuesday and the data team on Thursday, the 38% window is built into it.
Canaries that page on persistence, not noise
In production, evaluation runs in 30-minute windows that cycle through the golden set over about 24 hours, with a full daily run as the authoritative scorecard. The team tried 15 and 60 minutes before settling on 30. On-call is paged only after three consecutive breaching windows, because single-window alerts eroded trust.
Canaries catch what pre-production cannot: an upstream table changes its definition, and an agent that passed every gate yesterday is wrong today. It is the same reason an agent can return a 200 and still be wrong. The golden set is fed from three places: sampled production queries, including low-scoring live sessions; regression cases from past failures; and synthetic edge cases. Every failure becomes a test.
What the numbers say, and what they do not
AWS reports its pass rate rising from 93% to a stable 96% over three months, pure hallucinations falling from 138 to under 52 per 10,000 responses, unauthorized calculations falling from 59 to under 44, and mean time to detect dropping from days or weeks to under four hours. These are AWS’s figures for its own system, and the post marks several of its charts as illustrative.
Read the 96% carefully: four answers in a hundred still fail a check. The pipeline does not make the agent right; it makes errors visible within hours and turns each one into a test. For an analytics assistant whose answers a seller reads before acting, that can be acceptable. For an agent that writes to a system of record, the same pass rate needs a different control in front of the write: approval.
None of this requires AgentCore. It requires someone who owns “correct” as an operational metric, the way someone already owns uptime, and a release process that refuses to ship half an agent.