When coding agents keep misusing your API, test your docs. Microsoft's AX playbook shows how

Picture an internal platform team that ships version 3 of its data access SDK. A month later, half the new code across the company still calls the version 2 method it deprecated. Nobody wrote that code by hand. Coding agents did, because everything they could read, from old examples to forum answers to a stale page in the internal docs, pointed them to the old way.

The usual reaction is to blame the model and wait for a better one. Microsoft’s developer relations team suggests a different reaction: evaluate the agent’s experience of your technology the way you once evaluated a developer’s.

On October 9, Waldek Mastykarz, a principal developer advocate at Microsoft, published the Agent Experience (AX) Practitioner Playbook, “our method for evaluating what AI coding agents do with your technology.” It grew out of evaluations of Azure, Azure Cosmos DB, the SharePoint Framework and Microsoft 365 Copilot extensibility since fall 2025.

Find where it broke

The playbook’s most useful idea is that “the agent got it wrong” is not a diagnosis. Its trajectory analysis separates failures that look identical in the final code: an extension that never loaded, one that loaded but was never called, and one that was called but applied incorrectly. The playbook names nine recurring failure patterns.

1. Skill, extension or doc exists
2. The agent loads it
3. The agent calls it for the task
4. The agent applies it correctly
5. The code runs and meets criteria

What you publishWhat the trajectory shows

Each failure has a different owner. If the extension never loaded, the problem is packaging or discovery. If it loaded but was not called, its description did not match how the agent framed the task. If it was called and misapplied, the documentation or the example is wrong. Better models may reduce some of these errors. None of them fixes a stale example.

Fix the source, and prove the fix

The method closes the loop at the source. Proposed fixes to docs, skills or tool descriptions are tested as hypotheses against the same scenarios before shipping, and the evidence goes to the team that owns the product. Microsoft says this approach produced 46 closed improvements to the Azure Cosmos DB Agent Kit and dozens of shipped fixes to docs and agent extensions overall.

The evaluation model is just as deliberate. Results need criteria that judge meaning, plus gates that confirm the code actually runs. Criteria must be written so a judge scores them consistently, calibrated, and versioned as the product changes. That last point is easy to skip: an SDK that changes while its evaluation criteria stay the same will soon be graded against the wrong answer. It is the principle we described in regulated AI needs proof from outside the model: an LLM judge counts as evidence only once it has been calibrated.

A companion AX Practitioner skill helps people apply the method. Microsoft reports that across 330 questions, its answers averaged 95% agreement with the playbook. That figure measures fidelity to the playbook, as reported by its authors, not coding-agent accuracy.

Your internal platforms are the real audience

Microsoft wrote this for its own products, but the larger audience is inside enterprises. Every company with an internal developer platform, shared libraries, golden paths and API standards now has a second kind of consumer: the agents its developers use. Those agents read the internal wiki, the README files and the example repositories more literally than any person, and they repeat what they find at scale.

That makes agent experience a platform engineering responsibility with three practical consequences:

• Someone owns it. For each internal SDK or API, name the team responsible for how agents use it, the same way someone owns its developer documentation.
• Scenarios become regression tests. A small set of realistic agent tasks per platform, rerun when the platform changes, catches regressions in docs and examples before they spread through generated code.
• Old examples get retired on purpose. Deprecated patterns in examples and docs are now an active source of defects, because agents retrieve them.

We made a related point when Google turned its developer docs into an agent API: documentation is becoming agent context. Microsoft’s playbook adds the discipline that should come with it, measuring whether that context actually works. And because what agents produce enters the software supply chain, as we described in GitLab’s governed software factory, a defect in your docs can become a defect in production code within days.

The playbook is a method, not a product, and its results are Microsoft’s own. It is still the clearest public description of a job most platform teams have not noticed they now have.