Meta reports a 60% true-positive rate for LLM-generated tests built to fail on risky diffs

Meta says its Catching Just-in-Time Tests system is now part of its infrastructure. It runs overnight on spare capacity against the previous day's diffs that a risk model flags. One workflow infers a change's intent and builds tests aimed at the ways that intent could go wrong. A second treats every diff as a bug and generates tests that expose any behavioral difference. Tests that fail on the change are candidates, which an ensemble of LLM judges and a rule-based filter called RubFake rank before anyone is contacted.

Meta's figures: 589 engineers, 64% of those contacted, engaged with the results, and they reported a 60% rate of true positives. In the first half of 2026 the system played "a key role" in catching 74 serious privacy bugs at the diff stage, and Meta says privacy incidents of that severity fell 20% over the same period. Meta calls the review model human-on-the-loop: engineers set criteria and "review outcomes at an aggregate level."

The page text we read does not say how the 20% figure was derived or what it was compared against. The 60% rate is engineer-reported, and we did not open the accompanying paper.

Why it matters: This is a verification stage built around executable evidence: a test that fails on the diff, rather than a model's opinion of it. Meta's pipeline gives two of its five stages to filtering weak catches before a person sees them. If you add a similar check to an agent loop, the filter and the reporting threshold are the parts to design first.