OpenAI's COBOL migration benchmark is close to saturated. That does not mean banks can retire the mainframe
OpenAI reports 97.9% on its own COBOL migration evaluation and big gains in COBOL, Fortran and Visual Basic. The useful part is how it defines success: byte-for-byte behavior, not code that compiles.
Listen to this article · 6 min
AI-generated narration of the full article.
The most important sentence in OpenAI’s October 19 research post is not a score. It is an example about rounding. COBOL’s fixed-point arithmetic discards excess decimal digits unless the program explicitly asks for rounding. A rewrite in Java or Python that rounds those values instead will compile, run without errors and pass a casual review. It will also produce different financial results, a fraction of a cent at a time, across millions of transactions.
That is the whole problem of legacy modernization in one line, and it is why the way OpenAI measured its models matters more than the headline number.
Two evaluations, two different skills
OpenAI built two evaluations for what it calls low-resource programming languages, the ones with little public code to learn from but a great deal of critical infrastructure behind them. OpenAI cites estimates that COBOL supports more than 40% of online banking systems and 95% of ATM transactions.
MultiLangBench asks a model to build a SQLite-compatible database engine from scratch in one of 20 languages, over an extended coding session. Tests adapted from SQLite’s own suite check the resulting database files. At maximum reasoning effort, GPT-6 Astra reaches an average pass rate of 87.4%, against 26.8% for GPT-5.6 Sol. COBOL is still the hardest language, at 72.1%, up from 10.9%. OpenAI released the evaluation setup and test suite on GitHub.
The COBOL migration evaluation asks a model to translate existing COBOL programs into C++, Java, Kotlin, Python or Rust while preserving what they do. Hidden tests require byte-for-byte parity on observable outputs, and a migration counts only if every test passes. GPT-6.1 Sol reaches 86.6% at low reasoning effort, against 5.4% for GPT-5.6 Sol at the same setting, at 61% of the API cost. At higher effort it reaches 97.9%, which OpenAI itself describes as close to saturating the evaluation.
These are different tasks with different models. The 97.9% is about migration; the 87.4% is about building new software. Neither is an independent production result.
Why the success criterion is the news
Most demonstrations of AI-assisted modernization measure whether the new code compiles, whether it looks idiomatic, or whether a reviewer likes it. OpenAI’s migration test measures whether the program still does exactly what it did, down to the byte. That is the right definition, and it is the same one behind Google’s Dual Run approach: the replacement is proven by replaying real behavior against the original, not by reading the new code.
A near-saturated evaluation also tells you something useful: translation of self-contained programs is no longer the bottleneck. The cost and risk of a migration move to everything around the translation.
| Question | OpenAI's evaluation | A production mainframe migration |
|---|---|---|
| Unit of work | Individual COBOL programs | Thousands of programs, copybooks and JCL jobs |
| What defines correct | Hidden tests, byte-for-byte output | Business rules nobody wrote down, plus regulators' expectations |
| Dependencies | Contained in the test harness | CICS, DB2, VSAM files, batch schedules, downstream feeds |
| Evidence of equivalence | All tests pass | Parallel runs on production data, reconciled |
| Who signs off | The evaluation | Business owners, audit and the system's operators |
What 97.9% does not tell a bank
Where the tests come from. In OpenAI’s evaluation, someone wrote the hidden tests that define correct behavior. In a 40-year-old payments system, that definition is the hard part. Business rules live in the code, in operators’ habits and in downstream systems that depend on quirks. As AWS’s Azure-to-AWS conversion showed in a different setting, swapping the visible calls is easy; the semantics hide in the edges.
What the harness leaves out. The evaluation cannot represent every external dependency: transaction monitors, file layouts, batch timing, the report a regulator receives every month. OpenAI says so directly: production use still requires expert review and testing against the system’s business rules and dependencies.
Who designed the test. These are OpenAI’s evaluations of OpenAI’s models. The released MultiLangBench suite lets anyone inspect and rerun the database task; the migration evaluation’s hidden tests are, by design, not something you can audit from outside.
How to use the result
The practical change is in the economics of the program, not in its governance. If translation becomes cheap and accurate, the budget should move to the two activities that decide whether a migration succeeds: discovering behavior (instrumenting the current system, capturing inputs and outputs, mining the rules) and proving equivalence (parallel runs, reconciliation, sign-off by the people who own the outcomes).
A useful test for any vendor pitching AI-led modernization: ask them to show the equivalence harness before the translation demo. If the harness does not exist, the 97.9% belongs to someone else’s programs.