(Credit: BlackJack3D via Getty Images)
An AI program has climbed the leaderboard for discovering real-world software vulnerabilities, besting the human reviewers to take the top spot in the US.
The program, called Xbow, currently ranks number one on the US-based leaderboard at HackerOne, a platform that coordinates software vulnerability discoveries with major companies.
Over the last few months, Xbow has increased its reputation score at HackerOne by reporting over 1,000 apparent software flaws; 132 have been reported as officially discovered and resolved. Impacted companies include The Walt Disney Company, AT&T, Ford and Epic Games.
(Credit: HackerOne)
(Credit: Xbow)In total, the AI program has submitted nearly 1,060 vulnerabilities, the startup behind Xbow announced on Tuesday. “All findings were fully automated,” the team added, "though our security team reviewed them pre-submission to comply with HackerOne’s policy on automated tools."
While 132 of the flaws were officially resolved, another 303 were classified as “triaged,” meaning the reported bug has been acknowledged, but not resolved. Another 125 are pending review.
So, it’s possible Xbow may have discovered an even larger crop of vulnerabilities that still need to be confirmed. But the AI program didn’t always find a new security issue; 208 of the submitted reports were marked as “duplicates” while another 209 were flagged as merely “informative.” The remaining 36 were declared not applicable.
Still, the results show how new AI programs could shake up the cybersecurity industry through automated vulnerability discovery at a scale that outpaces human security researchers. “Notably, around 45% of Xbow’s findings are still awaiting resolution, highlighting the volume and impact of the submissions across live targets,” the Xbow team adds.
(Credit: Xbow)In addition, technology promises to help companies stay ahead of malicious hackers who have also been trying to adopt generative AI. Xbow is designed to be fully autonomous, with the ability to complete “comprehensive penetration tests in just a few hours,” the company says.
But Xbow is also raising some concerns about whether it’s generating quantity over quality in terms of vulnerability reports. “Receiving hundreds of AI-generated bug reports would be so demoralizing and probably turn me off from maintaining an open source project forever,” wrote one user on the Hacker News forum. “I think developers are going to eventually need tools to filter out slop.”
Brendan Dolan-Gavitt, an Xbow AI researcher, responded to the skepticism and criticism, writing: “The main difference is that all of the vulnerabilities reported here are real, many quite critical.” Others also point out the submissions from human security researchers on HackerOne can also be of low-quality.
HackerOne also chimed in on Xbow's development: "AI is a force multiplier for crowdsourced security,” said Michiel Prins, HackerOne's cofounder, in a statement. “Hackbot companies like Xbow are bringing impressive innovation to the space, accelerating how we discover and respond to vulnerabilities. But AI doesn’t learn to hack on its own—hackers train it. Human researchers remain essential partners in this feedback loop, and while AI leads in volume, we still see humans delivering the findings with the greatest business impact. Hackbots are simply the next step in an evolution driven by human ingenuity harnessing automation.”
Xbow released the results as it's trying to sell its technology to customers. Bloomberg reports that Xbox recently raised $75 million through a new funding round.