The development

What happened

Collinear AI described CWE-bench v2, a benchmark for AI agents auditing and patching real open-source code. Its 100 defensive security tasks cover 91 Common Weakness Enumeration types and require agents to fix a weakness while preserving legitimate behavior.

Key facts

  • The authors say every task was reviewed by hand and its programmatic checks were curated with help from LLM judge panels. This is reported task-design evidence, not a TideReach audit.

  • The documented four-rollout setup reports mean reward over 400 trials. An average across attempts is not the same statistic as solving a task at least once in four attempts.

  • Dataset files are gated behind access conditions. A separate held-out evaluation set uses the publisher’s contact process; neither was accessed for this coverage.

The context

Why it matters

The benchmark makes retained functionality part of defensive success rather than treating removal of one exploit as sufficient. Results remain specific to selected tasks and checks; they do not establish security for arbitrary production repositories. The publisher reports a model leaderboard; TideReach has not independently reproduced its results.