Three in four AI-generated vulnerability patches leave something broken

Off-by-1 Labs, 1Password's new security research group, graded 6,080 patches from ChatGPT 5.5 and Claude Opus 4.8 against six freshly disclosed CVEs: about one in four fully closed the flaw, and roughly one in twenty opened a new one. More than a third of the apparent fixes were fragile — blocking the demonstrated exploit with a narrow check while leaving the vulnerable code reachable — and none of that looks wrong on review or in a passing test. Fix direction mattered more than model or harness: a correct direction in the prompt closed the bug about two thirds of the time, a plausible but wrong one about one in six, so handing a patching agent an unverified diagnosis is worse than handing it nothing.

Source: helpnetsecurity.com ↗