Your agent's score can change overnight while the agent stays the same. Snowflake's judge switch shows why

Picture a dashboard that tracks an agent’s answer correctness week by week. On Wednesday the line moves. Nobody shipped a prompt change, the model behind the agent is the same, the data has not changed. The ruler did.

That is the situation Snowflake describes in a pending change to Cortex Agent and Cortex Analyst evaluations. The judge model behind the original metrics, Claude 4 Sonnet, “reaches end-of-life on October 14, 2026.” Snowflake therefore expects to switch the defaults from October 13, with the usual caveat that dates may change.

What changes, and for whom

The change applies to metrics that leave their version or judge model unset or set to auto:

• System metrics (answer correctness, logical consistency, tool selection accuracy and tool execution accuracy for agents, and SQL correctness for Analyst) move from metric version v1 to v3.
• Custom metrics move from claude-4-sonnet to claude-sonnet-4-6 as the judge, falling back to openai-gpt-5.4 if the preferred model is not allowed or available for the account.

Snowflake is explicit about the consequence: “Scores from different metric versions aren’t directly comparable.” Judge-based scores can shift, and evaluation costs may change because judge models carry different credit rates. One metric is unaffected: tool selection accuracy does not use a judge model.

Pinned configurations do not move. But a pin to the retiring judge is not a safe harbor: “Metrics pinned to v1 or claude-4-sonnet stop working at end-of-life.” Accounts that had not used v1 before August 12, when it entered a legacy state, could not run it at all.

What happens to each configuration

BreaksYou decide

1. Pinned to v1Stops working at end-of-life
2. Unset or autoMoves to v3 and the new judge without notice
3. Pinned to v2Stable until you choose to move
4. Pinned to v3New default, recalibrated thresholds

The evaluation stack is a dependency, not a constant

Most teams version their agent: prompt, model, tools, retrieval configuration. Far fewer version the thing that grades the agent. Yet an LLM-as-judge metric has its own model, its own prompt and its own definition of a good answer, and any of them can change underneath you. When it does, three things break quietly:

• Trend lines. A quality chart that spans the switch compares two different measurements. Any improvement or regression across that date is uninterpretable.
• Gates. A CI/CD check that blocks a release below a score threshold may start failing good releases or passing bad ones, because the threshold was calibrated against the old judge.
• Comparisons. Results from two teams, or two platforms, are comparable only if both used the same judge and metric version, the problem we raised when Collibra began showing agent metrics from four platforms on one dashboard.

This is the evaluation side of a pattern we described in model retirement as the new API deprecation: when a model retires, behavior changes even if no code does. Here the retiring model is the grader.

What Snowflake recommends, and what to add

Snowflake’s preparation steps are sound: test system metrics with version: "v3" and recalibrate thresholds in dashboards, alerts and CI/CD; pin a metric version (for example v2) or a custom judge model if you want to control timing; and confirm that the new judge models are allowed for your account and for the role that runs evaluations, and reachable through your cross-region inference settings. The new judges have context windows of about one million tokens, which Snowflake says reduces failures on long agent traces.

Beyond this specific change, the general practice is worth adopting on any platform:

1. Pin evaluation configurations explicitly and record the metric version and judge model with every score you store.
2. Run old and new side by side on a fixed evaluation set before switching, and record the offset between them.
3. Mark the switch on every chart, so nobody reads a measurement change as a quality change.
4. Recalibrate against people. A new judge should be checked against human-labeled examples, the calibration step we argued makes evaluation the control surface rather than decoration.

If your agent’s score moves this week, check the judge before you check the agent.