Key Points:
-
Major Finding: OpenAI researchers have determined that even the most advanced AI models cannot solve the majority of coding problems presented to them.
-
Comparison to Human Coders: The findings suggest that current AI capabilities remain inferior to those of human coders, despite claims from OpenAI’s CEO, Sam Altman, that AI would surpass “low-level” software engineers by the end of the year.
-
Benchmark Introduction: The research utilized a new benchmark called SWE-Lancer, which comprises over 1,400 software engineering tasks sourced from Upwork.
-
Models Tested: The study tested three large language models, including OpenAI’s own o1 reasoning model, the flagship GPT-4o, and Anthropic’s Claude 3.5 Sonnet.
Executive Summary:
Recent research from OpenAI reveals that even the top AI models fall short in effectively solving a significant number of coding tasks. This research challenges the notion that AI can outperform human programmers, with CEO Sam Altman claiming a potential for AI to overtake lower-tier engineers by year-end. Utilizing the newly established SWE-Lancer benchmark with a comprehensive set of coding challenges, the findings illuminate the current limitations of AI in software development.
12ft.io Link: https://12ft.io/https://futurism.com/openai-researchers-coding-fail
Archive.org Link: https://web.archive.org/web/https://futurism.com/openai-researchers-coding-fail
Original Link: https://futurism.com/openai-researchers-coding-fail
User Message: OpenAI Researchers Find That Even the Best AI Is "Unable To Solve the Majority" of Coding Problems
for more on see the post on bypassing methods