AI security scanners find more problems. GitHub now lets enterprises see whether developers fix them

There is an easy way to make an AI security scanner look successful: count what it finds. More findings per pull request, more repositories covered, more vulnerabilities “caught.” None of those numbers says whether the code is any safer. A finding nobody fixes is a notification, and a stream of them teaches developers to stop reading.

GitHub’s October 13 update moves the measurement one step closer to what matters. Organization and enterprise administrators now get metrics for AI Scan in pull requests inside the security overview: findings generated and resolved, remediation rate, which repositories generate findings, trends over the reporting period and how often developers apply the autofix suggestions. The page formerly called “CodeQL pull requests” becomes “Code scanning on pull requests,” with separate tabs for CodeQL and AI Scan. The AI Scan tab shows total findings and how many came with or without an autofix suggestion.

The feature is in public preview, on github.com including GitHub Enterprise Cloud, for customers with GitHub Code Security or GitHub Advanced Security. GitHub Enterprise Server is not supported in this release.

Why separating AI Scan from CodeQL matters

CodeQL is a rule-based analysis engine; its findings are reproducible and its false positive patterns are well understood. AI Scan uses a model to look for problems in a pull request. Mixing the two in one dashboard would blur exactly the comparison an enterprise needs: does the AI scanner add findings developers act on, or noise they dismiss? With separate tabs, a security team can ask that question per repository.

That is the right way to evaluate any AI system in the delivery pipeline: as its own component with its own metrics, the approach we argued for when we wrote that retrieval needs to be measured as its own system.

What the numbers can tell you

Used carefully, the new metrics support three decisions:

• Rollout. Remediation rate by repository shows where AI Scan earns attention and where it is ignored. Low-remediation repositories need either tuning or a conversation with the team.
• Autofix trust. How often developers apply suggested fixes is a direct signal of whether the fixes are good enough to use. A low rate means review cost without benefit.
• Coverage gaps. Repositories that generate no findings at all are worth a look: they may be clean, or the scan may not be running where it should.

What they cannot tell you

A higher remediation rate does not prove lower risk, for reasons worth writing into any report that cites these numbers:

• A fix closes a finding, not a vulnerability. Applying an autofix resolves the alert. Whether the fix is correct, complete and free of side effects is a separate question that needs review or testing, especially for AI-suggested code.
• Resolved needs a definition. The changelog does not say whether a dismissed finding counts as resolved. Check the metric definitions before reporting the number upward.
• Severity is missing from the headline. One fixed critical issue matters more than fifty fixed style warnings. Report remediation by severity, not in aggregate.
• Missed issues are invisible. The dashboard measures what the scanner found. It says nothing about what it missed, which only independent testing can estimate.

This is the same trap we described in hours saved are not value: an activity metric presented as an outcome. Findings fixed is better than findings found. Risk reduced is better still, and it has to be measured with something other than the scanner itself.

The practical takeaway

If you run AI Scan, use the preview to set a baseline now: remediation rate and autofix acceptance by repository, split by severity. Pair it with a periodic sample of applied autofixes reviewed by a security engineer, so you know whether accepted fixes are correct. Google’s experience with agents in code review showed that AI security tools earn their place when their output lands where developers already decide. GitHub’s metrics now let you check whether that is happening.