The development
What happened
OpenAI’s 19 October report introduces two evaluations of AI coding models: MultiLangBench and a separate COBOL migration test. They assess building software across less-represented languages and preserving existing behavior, rather than merely producing code that compiles.
Key facts
MultiLangBench asks models to build SQLite-compatible engines across 20 languages, checking database contents against reference SQLite. Its setup and tests are released on GitHub.
OpenAI reports GPT-6 Astra averaging 87.4% test passes versus GPT-5.6 Sol’s 26.8%; GPT-6 Luna reaches 40.3% at approximately 78% lower estimated API cost.
COBOL migrations target C++, Java, Kotlin, Python or Rust and must pass every hidden test with byte-for-byte output parity.
GPT-6.1 Sol scores 86.6% at low reasoning effort, versus 5.4% for GPT-5.6 Sol at that setting; higher effort reaches 97.9%.
The context
Why it matters
Behavior-preserving tests can expose differences such as financial rounding errors that compilation misses. These vendor-run scores are task-specific, not general correctness rates; production migration still requires expert review and business-rule and dependency testing.