(Credit: Sammyvision via Getty Images)
Much has been made of how AI tools like Cursor Pro or Anthropic's Claude will revolutionize coders' day-to-day experience, potentially shaving tens of hours off their working weeks—but research has now cast doubt on those assumptions.
A new study assessing the performance of 16 experienced software developers working on 246 different tasks found that using AI tools actually increased completion time for the tasks by 19%. This flew in the face of the expectations of researchers, who had anticipated that the AI-assisted coders would be 24% faster than the non-AI control group.
The study, first spotted by The Register, came from Metr, a nonprofit AI-focused research organization.
Reasons why AI slowed down developers included low reliability. The study found that the developers accepted fewer than 44% of the AI's code suggestions, and the majority reported making major changes to clean up the AI-generated code. In fact, the developers spent 9% of their time reviewing or cleaning the AI outputs.
Though the AI-assisted developers did save a lot of time spent actively coding or researching, many of these gains were nullified by time spent crafting prompts, waiting for the AI to come back with responses, or proofing its output for errors. In addition, the study found that participants consistently overestimated AI's ability to assist at the task at hand, even after hours of using the tool.
However, there are plenty of caveats to this research. The participants all had five years of coding experience, with only "moderate" AI experience. So results may have been different for less experienced coders.
The participants were also set to work on very large and complex existing code repositories, which broadly “had very high quality bars for code contributions,” according to the researchers. So results may be different for a completely green coder working on a simple app.
This isn't the first time that we've heard rumblings about AI causing as many problems as it solves in the workplace. Research from economists at the University of Chicago and the University of Copenhagen released earlier this year found that AI has yet to move the needle in terms of real-world productivity. Though it caused gains of roughly 3% in overall worker productivity, this was offset by the workload of tasks created by AI, such as teachers checking homework for ChatGPT use.