Gemini 4 Argon Leads CWE-bench v2 Cybersecurity Benchmark
CWE-bench v2 opens 100 security repair tasks across 91 CWEs, with Gemini 4 Argon leading 11 evaluated models at 67.8% pass@1 and 81.0% pass@4, in testing.
Summary
On November 3, 2026, Collinear AI released CWE-bench v2, a gated public Hugging Face benchmark containing 100 real open-source repository tasks across 91 CWEs. Agents receive a checkout and realistic instruction, then must find and repair a security weakness without breaking existing functionality. Each self-contained Harbor task packages a Dockerized environment and deterministic verifier, enabling reproducible programmatic scoring without extra cost. LLM judge panels and manual reviews check fairness, realistic instructions, behavioral grading, failing starter code, passing reference fixes, regression safety and resistance to memorized upstream patches. A separate held-out set tracks unseen-task progress through cyber@collinear.ai.
Among 11 evaluated models, Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4, narrowly ahead of Claude Opus 5.5 at 67.3% and 78.0%. At least one model solves 95 tasks, yet Gemini fails 19% on every attempt. The cost frontier includes DeepSeek V4.1 Flash (1017) at 57.0% pass@1 and $0.04 per rollout, Opus at $1.83 and Gemini at $3.92. Open-weight results update through model-repository pull requests, while a separate Space includes API-only Gemini, Claude and GPT models. The open set, grader, harness and registry.json run directly through Harbor after users accept access terms and save an HF token. Using -k 4 makes four attempts per task and averages 400 trials; rl-environment tags also enable defensive-agent training.
Positives
- 100 public tasks spanning 91 CWEs broaden testing across many software vulnerability classes.
- 95 of 100 tasks were solved by at least one of the 11 evaluated models, confirming broad solvability.
- Deterministic verifiers make scores reproducible without additional evaluation costs.
- DeepSeek V4.1 Flash (1017) achieves 57.0% pass@1 for just $0.04 per rollout.
- Open graders, harnesses and tasks allow independent evaluation, feedback and public contributions.
- rl-environment tags let the same tasks provide rewards for training defensive agents.
Risks & concerns
- Gemini 4 Argon still fails 19% of the benchmark on every attempt despite leading overall.
- Claude Opus 5.5 costs $1.83 per rollout and Gemini 4 Argon costs $3.92, far above DeepSeek's $0.04.
- The dataset is gated, requiring acceptance of access conditions and Hugging Face credential configuration.
- The held-out benchmark is not publicly runnable and requires contacting Collinear AI.
- Five tasks remain unsolved by every evaluated model.