The development

What happened

OpenAI’s 19 October report introduces two evaluations of AI coding models: MultiLangBench and a separate COBOL migration test. They assess building software across less-represented languages and preserving existing behavior, rather than merely producing code that compiles.

Key facts

  • MultiLangBench asks models to build SQLite-compatible engines across 20 languages, checking database contents against reference SQLite. Its setup and tests are released on GitHub.

  • OpenAI reports GPT-6 Astra averaging 87.4% test passes versus GPT-5.6 Sol’s 26.8%; GPT-6 Luna reaches 40.3% at approximately 78% lower estimated API cost.

  • COBOL migrations target C++, Java, Kotlin, Python or Rust and must pass every hidden test with byte-for-byte output parity.

  • GPT-6.1 Sol scores 86.6% at low reasoning effort, versus 5.4% for GPT-5.6 Sol at that setting; higher effort reaches 97.9%.

The context

Why it matters

Behavior-preserving tests can expose differences such as financial rounding errors that compilation misses. These vendor-run scores are task-specific, not general correctness rates; production migration still requires expert review and business-rule and dependency testing.