AI might write more code than ever, but when it comes to fixing bugs, it's still getting schooled by human developers. A fresh study from Microsoft Research put nine state-of-the-art AI models—from big names like OpenAI and Anthropic—to the test, giving them 300 real-world debugging tasks. Spoiler alert: they mostly flunked. Even with powerful tools and fancy prompts, the best performer, Anthropic’s Claude 3.7 Sonnet, only succeeded around half the time (48.4%, to be precise). The culprit? These models aren't great at using tools or understanding how programmers step through debugging—largely because they haven’t been trained with enough real debugging process data. So, while AI might be helping out at tech giants like Google and Meta, don’t expect it to replace engineers just yet. As the study puts it, AI needs “specialized data” like debugging traces to level up—and for now, it’s still the junior dev in the room.

Key Points:

• Microsoft Research evaluated 9 top AI models on their ability to debug software using a benchmark called SWE-bench Lite.
• Models like Anthropic’s Claude 3.7 Sonnet, OpenAI’s o1, and o3-mini were tested with access to tools like a Python debugger.
• Claude 3.7 Sonnet was the top performer with a 48.4% success rate. OpenAI's o1 and o3-mini trailed at 30.2% and 22.1%, respectively.
• Even the strongest models failed to debug most tasks, mainly due to limited understanding of debugging tools and processes.
• "There’s not enough data representing ‘sequential decision-making processes’—that is, human debugging traces—in current models’ training data," according to the study.
• The researchers suggest that fine-tuning with specialized data, like recorded debugger interactions, could improve AI capability.
• Other studies echo these findings: recent research found tools like Devin could only pass 3 out of 20 programming tests.
• Despite AI-generated code now accounting for a quarter of new code at Google, leaders like Bill Gates and IBM’s Arvind Krishna argue that programming jobs aren’t going anywhere.
• Takeaway: AI is a helpful programming assistant, not your lead software engineer—at least, not yet.
  Link to Article