OpenAI Enhances Coding In Low-Resource Languages
OpenAI's latest models improve coding in languages like COBOL and Fortran, which are crucial for essential services. This advancement may help teams maintain and modernize critical software.
Original reporting · openai.com
Original: “Advancing Coding in Low-Resource Programming Languages”
Read “Advancing Coding in Low-Resource Programming Languages” at OpenAI →
How this was written. Content Memory wrote this article with a language model from OpenAI’s reporting (model: @cf/meta/llama-3.3-70b-instruct-fp8-fast); it describes that reporting rather than adding to it. The original, linked above, is the full story.
Key points from OpenAI’s report
- OpenAI's latest models improve coding capabilities in low-resource programming languages like COBOL and Fortran.
- The company's evaluations, MultiLangBench and COBOL migration, assess a model's ability to build new software and preserve existing software's behavior.
- GPT-6 Astra and GPT-6 Luna showed significant improvements in both evaluations, with GPT-6 Astra achieving an average test pass rate of 87.4% across 20 programming languages.
According to OpenAI, certain programming languages lack sufficient code and documentation, making it challenging for models to learn from them. Despite this, these languages support vital infrastructure and services worldwide.
OpenAI reported that languages like COBOL and Fortran are used in critical areas such as online banking, weather forecasting, and nuclear record safety analysis. As a result, improving AI's ability to work with these languages is essential.
The company's latest model advancements were guided by feedback from developers and enterprise customers who struggled with coding models in low-resource languages. Maintaining and modernizing software in these languages can be difficult due to incomplete documentation and the absence of original developers.
To measure progress, OpenAI developed two evaluations: MultiLangBench and a COBOL migration evaluation. These tests assess a model's ability to build new software and preserve existing software's behavior across different programming languages.
The MultiLangBench evaluation requires a model to build a SQLite-compatible database engine from scratch in a specified programming language. This test evaluates a model's ability to organize a codebase, manage state, handle errors, and add new features while preserving existing functionality.
OpenAI's latest models, including GPT-6 Astra and GPT-6 Luna, showed significant improvements in both evaluations. GPT-6 Astra achieved an average test pass rate of 87.4% across 20 programming languages, while GPT-6 Luna reached 40.3% at a lower estimated API cost.
The COBOL migration evaluation tests a model's ability to migrate existing COBOL programs to other languages while preserving their behavior. GPT-6.1 Sol achieved 86.6% accuracy at low reasoning effort, compared to 5.4% for GPT-5.6 Sol, and reached 97.9% accuracy at higher reasoning efforts.
Questions and answers
What programming languages are considered low-resource?
Languages like COBOL, Fortran, and Visual Basic are considered low-resource due to limited code and documentation available for models to learn from.
Why is it important to improve AI's ability to work with low-resource languages?
Improving AI's ability to work with these languages may help teams maintain and modernize critical software that supports vital infrastructure and services worldwide.
How can developers access OpenAI's latest models?
Developers can try OpenAI's latest models through the API on coding tasks in the languages their team uses.
