OpenAI reports COBOL migration accuracy rising from 5.4% to 86.6% on its own test

OpenAI built two evaluations. MultiLangBench asks a model to write a SQLite-compatible database engine from scratch in one of 20 languages, scored against tests adapted from SQLite's own suite. On it, GPT-6 Astra averages an 87.4% pass rate at maximum reasoning effort, against 26.8% for GPT-5.6 Sol. COBOL moves from 10.9% to 72.1% and Fortran from 21.4% to 83.5%. The cheaper GPT-6 Luna reaches 40.3% at an estimated $0.77 per sample, compared with $3.48 for GPT-5.6 Sol.

The second test migrates COBOL programs to C++, Java, Kotlin, Python or Rust, graded on hidden tests that require byte-for-byte parity on observable outputs. A program counts only if every test passes. GPT-6.1 Sol scores 86.6% at low reasoning effort, against 5.4% for GPT-5.6 Sol, at 61% of the API cost. At higher effort it reaches 97.9%, which OpenAI says brings the evaluation close to saturation. The post's example of what the tests catch: COBOL's fixed-point arithmetic drops extra decimal digits unless rounding is requested, so a rewrite that rounds can run cleanly and still produce different financial results. OpenAI also says production use still needs expert review and testing against the system's business rules.

OpenAI says it is releasing the MultiLangBench setup and test suite on GitHub. We have not opened the repository, and the post does not say the migration test is released. All results are OpenAI's own, on its own models, with no outside replication reported. The post was at 41 points on Hacker News about two hours after it was submitted.

Why it matters: If you hold a legacy migration, the useful part for now is the grading method: hidden tests that demand identical observable output, with success meaning every test passes. That is a verification loop you can copy for your own port whatever model you use. Treat the 86.6% as a vendor's figure on a test it designed until someone outside runs it.