The development
What happened
Meta says it has deployed Catching JiTTests in production to generate executable tests that expose behavioral changes in proposed code. In an engineering post, the company reported internal results on engineer engagement, finding accuracy and privacy bugs; no independent reproduction is provided in the supplied evidence.
Key facts
Meta says production deployment followed five months of rollout work. The system runs overnight on spare capacity and prioritizes changes classified as risky.
One workflow infers intended changes and potential failure modes; another treats changes as mutations and tests for behavioral differences.
Multiple LLM judges and the deterministic RubFake assessor filter candidate failures before selective outreach to engineers. Engineers set criteria and review aggregate outcomes rather than every generated output.
Meta reports that 589 engineers engaged, representing 64% of outreach, and that 60% of reported findings were true positives.
Meta says the system helped catch 74 serious privacy bugs before production and reduced serious privacy incidents by 20% in the first half of 2026.
The context
Why it matters
Executable tests could help engineers identify consequential changes before code ships without reviewing every generated output. The reported impact remains uncertain outside Meta: its results are internal, and the evidence does not establish a universal performance improvement or safety guarantee.