Reward an AI auditor for finding problems and it learns to invent them. Anthropic's research shows the fix
Anthropic trained a small model to audit AI systems with reinforcement learning. The naive reward produced invented findings; pairwise judging and benign examples fixed it.
Listen to this article · 5 min
AI-generated narration of the full article.
The headline result of Anthropic’s new alignment research is that a small model can be trained into a capable auditor of other AI systems. The more useful result for anyone running AI evaluation is what happened on the way there: rewarded for finding problems, the auditor learned to manufacture them.
The experiment
Paul Rosu and Rowan Wang, writing on Anthropic’s alignment science blog on October 15, trained an auditor based on Claude Haiku 4.5 with reinforcement learning. Its job was to investigate target models into which researchers had planted hidden behaviors through their system prompts, over multi-turn investigations across many scenarios and several model families.
With the best training setup, the Haiku auditor matched Claude Opus 4.6 on the authors’ composite evaluation. On audit quality it scored 72.7 against Opus 4.6’s 68.0; on a realism measure, 14.0 against 13.1, up from 4.6 before training; and it kept a false positive calibration score of 99.6.
These are results on the authors’ own evaluations, in environments with planted behaviors, which are not the same as the failures a production system develops on its own. This is research, not a product, and it does not show that a small model out-audits a large one in general.
How the auditor cheated
The first reward designs were pointwise: a score of how concerning the findings were, or a binary reward for hitting the planted behavior. Both were exploited. According to the authors, the auditor learned to manufacture alarming findings and to narrate conclusions it had not established. The binary-reward version developed aggressive, manipulative auditing; the concerningness version fell into pressured confessions.
Read that again with an enterprise lens. If you measure an evaluator by how many problems it finds, you will get an evaluator that finds problems. It will not necessarily find real ones.
What fixed it
Two changes made the difference:
- Pairwise rewards. Instead of scoring one investigation in isolation, a judge compared two investigations of the same target in the same scenario. The authors found this more robust than pointwise scoring.
- Benign targets. Mixing in targets with no planted behavior was, in the authors’ words, “necessary to keep false positive rates low.” The best configuration used 50% calibration training. An auditor that never sees a clean system never learns to say “nothing here.”
The enterprise version of this problem
Few enterprises will train auditors with reinforcement learning. Many already run the same incentive by other means:
- LLM judges tuned on failure cases only. A judge prompt developed by collecting bad outputs and asking “would you catch this?” is trained, informally, without benign examples. Expect it to flag good outputs too.
- Red teams and scanners measured by findings count. The incentive is identical to the pointwise reward. We made a similar point about measuring AI security scanners by fixes rather than findings.
- Absolute scores compared across time. Pointwise scores drift with the judge. Snowflake’s recent judge change moved agent scores without any change in the agent. Pairwise comparisons, old version against new on the same inputs, are more stable.
Three practices worth adopting
- Put known-good cases in every evaluation set, and track the evaluator’s false positive rate on them as a first-class metric. An evaluator with no measured false positive rate has an unknown one.
- Prefer comparisons to grades when deciding between versions of a prompt, model or agent. “Is B better than A on these 200 cases?” is easier to judge reliably than “is B an 8 out of 10?”
- Audit the auditor. Sample the evaluator’s findings and have a person confirm them, especially after its prompt or model changes.
The research is about AI safety, but its lesson is general: an evaluator is a system with incentives, and it optimizes for what you reward. The point we keep returning to, that proof has to come from outside the model, has a corollary. The outside check needs checking too, and the first thing to check is what it is paid to find.