Only 26% fully fixed the original vulnerability without changing how the software was intended to work.
Key Takeaways
- 53.9% of the tested AI-generated security patches failed to fix the original vulnerability, introduced a new security vulnerability, or did both.
- 4.5% introduced at least one new security vulnerability.
- Only 26% fully fixed the vulnerability without changing the software’s intended behavior.
1Password’s Off-by-1 Labs found that 53.9% of AI-generated security patches tested against complex software vulnerabilities failed to fix the original weakness, introduced a new security vulnerability, or did both.
The research paper found that 4.5% of the patches introduced at least one new security vulnerability. Some of those patches fixed the original problem, while others left it unresolved.
Only 26% fully fixed the vulnerability without changing how the software was intended to work.
The researchers concluded that AI-generated patches for complex vulnerabilities still require careful review by engineers who understand the affected software.
Researchers tested more than 6,000 patches
The researchers generated 6,480 patches using OpenAI’s ChatGPT 5.5 and Anthropic’s Claude Opus 4.8. They excluded 400 because the models tried to look up the software developers’ existing fixes rather than create their own. That left 6,080 patches in the reported results.
The tests covered six recently disclosed vulnerabilities in open-source software, including Linux, Google Chrome, Apache ActiveMQ, Exim, Spring AI and Google’s Gemini command-line tool.
The researchers selected complex vulnerabilities that required changes across multiple parts of the software. They also chose recently disclosed flaws to reduce the likelihood that the correct patches were included in the models’ training data.
The paper said it was not intended to compare which model performed better. Instead, the researchers used the two models to generate and review patches under different instructions and testing conditions.
Some patches changed software behavior
Another 20.1% of patches fixed the original vulnerability but changed how the software behaved.
For example, some patches blocked valid inputs along with malicious ones or changed how parts of the software processed information. The researchers classified these as fixes with unwanted software changes, rather than clean fixes.
The study also found that 37.5% of patches initially classified as fixes contained fragile security measures. These patches stopped the specific attack used in the test but did not remove the underlying weakness, meaning a different attack path could potentially expose it again.
The information supplied to the AI agent also affected its performance. When the researchers gave it the correct approach for fixing the vulnerability, 65% of patches succeeded. When they deliberately supplied a plausible but incorrect approach, the success rate fell to 15.2%.
Study relied partly on AI validation
AI models performed most of the initial patch reviews, with each model also reviewing patches created by the other. The researchers manually reviewed at least 10% of the patches in each test group and revised their grading rules when they found recurring errors.
The paper said the tested vulnerabilities were likely harder to fix than the average software flaw. It also described the results as a snapshot of model performance. Performance could change as the models and their online services are updated.
Off-by-1 Labs released its testing software, generated patches and supporting data with the paper. The researchers said further work is needed to determine when AI-assisted patching can reliably help software teams and whether reviewing AI-generated patches saves time compared with patches written by engineers.
