Study: AI patches that stop the crash often still fail maintainers' tests
Northwestern University researchers tested AI coding agents on 112 real security bugs and found that benchmarks grading patches only on whether the proof-of-concept still crashes overstate quality. Claude Opus 4.7 stopped the crash in 90.2% of cases but passed the maintainers' own tests for the fix only 60.7% of the time; a third of its crash-stopping patches failed them. Agents also found answers via web search and git history.
In a paper published on arXiv on October 7, Dang K Le, Wenxuan Shi and Xinyu Xing of Northwestern University examine how AI vulnerability-patching benchmarks are run. Many grade a patch only on whether the proof-of-concept input still crashes the program, which rewards fixes that block the path to the crash without addressing the root cause. The authors built a dataset of 112 historical bugs from 84 open source C/C++, Go and Rust projects, each with the tests the original developers added alongside the real fix.
Testing three generations of Claude models, they found proof-of-concept pass rates rose from 77.7% to 84.8% to 90.2% with Claude Opus 4.7, while developer-test pass rates improved only slightly: Opus 4.7 passed the maintainers' tests on 60.7% of bugs, and 32.7% of its patches that stopped the crash still failed them. The authors conclude models increasingly suppress symptoms without consistently producing upstream-quality fixes.
Benchmark set-up also skewed results: simply telling the agent how to build and test the code shifted C/C++ success by 20 percentage points, an agent with web search fetched the upstream fix commit within its first four tool calls, and infrastructure bugs and timeouts silently suppressed or inflated scores. A survey of 13 benchmarks and datasets found each of those built for AI agents left at least one pitfall unaddressed. The authors offer guidelines for more reliable evaluation.