AI agent observability tools: what each one actually covers, and what none of them can tell you
Snowflake, AWS, Microsoft, Collibra and others now sell observability for AI agents. They cover different layers. A guide to the four layers, the tools in each and the questions to ask before buying.
Listen to this article · 7 min
AI-generated narration of the full article.
“Which tool should we use to monitor our AI agents?” is the question platform teams ask most often once a second agent goes into production. The honest answer is that no single product answers it, because “observability” for agents covers four different jobs, and the tools on the market each do one or two of them well.
This guide maps those layers, places the tools we have analyzed in each, and ends with the questions that decide whether a tool will hold up in production. It will be updated as the market moves.
Four layers, four different questions
1. Traces: what did the agent do? The record of each run: prompts, completions, retrieved documents, tool calls with their arguments, tokens and session identifiers. Without it, nothing else can be explained.
2. Evaluation: was it right? Checks on quality, on live traffic and before each release: deterministic tests where the right answer is known, model-based judges where it is not.
3. Inventory, usage and cost: what is running, who owns it, what does it cost? Which models and agents are deployed, by whom, with what traffic, spend and quota.
4. Cross-platform governance: can we compare and act across vendors? A layer that collects signals from several platforms and puts them next to ownership, risk assessments and lifecycle status.
Classic application monitoring covers none of the four. It tells you an agent answered and how fast. As we put it when Snowflake announced its product, an agent can return a 200 and still be wrong.
| Tool | Traces | Evaluation | Inventory and cost | Cross-platform |
|---|---|---|---|---|
| Snowflake Agent Observability (private preview) | Prompts, retrievals, tool calls, tokens | Online LLM judge, offline results | Cost per trace | OTLP from custom frameworks |
| Amazon CloudWatch Omni | Five agent frameworks, multicloud | Evaluation-driven workflow | Not described | AWS, Azure, Google Cloud telemetry |
| Amazon Bedrock AgentCore evaluations | Not the focus | Built-in and custom evaluators | No | No |
| Microsoft Foundry tenant dashboard (reference architecture) | No, by design | No | Inventory, usage, cost, quota | Azure only |
| Collibra Platform 2026.10 | No, reads platforms | Pass rates from each platform | Token counts | Databricks, Bedrock, Gemini, Foundry |
The tools, layer by layer
Snowflake Agent Observability is the most complete trace and evaluation design we have seen announced. It records prompts, completions, retrievals, tool calls, tokens and sessions, derives latency, errors and estimated cost from them, and runs online LLM-as-judge evaluations on production traffic. It follows the emerging OpenTelemetry GenAI semantic conventions and supports LangChain, the OpenAI Agents SDK and Anthropic’s agent SDK. It was announced for private preview, so scope and pricing can still change.
Amazon CloudWatch Omni is AWS’s AI-assisted observability experience. It spans AWS accounts, regions and other clouds, including Azure and Google Cloud, uses OpenTelemetry, and gives agents built on LangGraph, CrewAI, the OpenAI Agents SDK, the Vercel AI SDK and Strands an evaluation-driven workflow. On October 19 it expanded to eight more regions, including Frankfurt, London and São Paulo.
Amazon Bedrock AgentCore evaluations sit in the second layer. The most useful public account of using them is AWS’s own: its sales agents passed every operational check while reporting a $410.08 figure as “$410.08 Million”, and the fix was an evaluation pipeline built into every release, with deterministic checks for known answers and custom evaluators where the built-in ones did not cover the failures that cost money.
Microsoft’s tenant-wide Foundry dashboard is not a product but a reference architecture, and it owns the third layer cleanly: inventory from Azure Resource Graph, usage from Azure Monitor, cost from Cost Management and quota tracked separately, with no prompts or completions collected by design.
Collibra Platform 2026.10 works at the fourth layer. It reads monitor pass rates and token counts from agents on Databricks, AWS Bedrock, Gemini Enterprise Agent Platform and Azure AI Foundry and shows them beside governance data. Its own documentation carries the caveat that matters: what is collected, and how pass or fail is decided, differs by platform.
Operations agents, such as AWS DevOps Agent and Azure SRE Agent, consume this telemetry rather than produce it. They diagnose incidents from it, and their diagnosis is a starting point, not a finding.
Five questions that decide whether a tool holds up
Does it emit or accept OpenTelemetry GenAI traces? Instrumenting agents to an open convention means the traces can move to another backend later. It is the cheapest insurance against lock-in in this market.
Does it grade what the agent did, or what it said it did? A judge that reads the agent’s final message grades the agent’s narrative. Google found its judge failing runs that had passed, because it read the summary instead of the trajectory. Ask whether evaluations can score tool calls and their order.
What happens to scores when the judge changes? Judge models retire too. When Snowflake moved Cortex evaluations to a new judge, it said plainly that scores from different metric versions are not directly comparable. A tool should let you pin, version and calibrate judges, and show which judge produced each score.
Where do known answers get checked deterministically? Model judges are for questions without one right answer. Revenue figures, dates, entity matches and policy limits should be checked against the source, as AWS does.
Where does the telemetry live, and who can read it? Traces hold the prompts, documents and customer data the agent was trusted with. The observability store needs the same access controls, masking, retention and residency decisions as the source systems. For teams in the EU or Brazil, ask which region processes each feature, not only which region hosts the console.
Where to start
For a first production agent: instrument traces with OpenTelemetry GenAI conventions, write down the failure types that would cost money, and build deterministic checks for those before adding a judge. Add the inventory and cost view when the second team ships an agent. Add a cross-platform layer only when agents run on more than one vendor’s platform and someone has agreed on shared checks, because aggregating incomparable numbers produces a dashboard, not evidence.