SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark
SWE-Bench Pro is a widely used benchmark of long-horizon software engineering tasks for coding agents.
ProofPaper ↗
Key points
- As agents improve, they reach task issues earlier agents never touched and interact with the evaluation environment in ways the original harness did not anticipate.
- We present SWE-Bench Pro V2, a revised public split built to keep the benchmark accurate as capabilities advance.
- Agentic audits, expert adjudication and blind re-implementation repaired or retired 480 of the 731 tasks: 23 contracted engineers spent 404 hours, each working in a language they write professionally, and every revised instruction was verified by a second engineer solving the task from the text alone.
- We release the benchmark and distill lessons for maintaining agent benchmarks over time.
Sources (1)
- [1]SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding BenchmarkarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 08:26 PM
SWE-Bench Pro is a widely used benchmark of long-horizon software engineering tasks for coding agents.
As agents improve, they reach task issues earlier agents never touched and interact with the evaluation environment in ways the original harness did not anticipate.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 16, 2026Agents' Last Doubt: Knowing When to Act and When to Step Back in Tool-Using Agents
- Oct 16, 2026Pronouncing Hard Words: Phoneme Adaptation of a Pretrained Conversational Text to Speech Model
- Oct 16, 2026sparqlite: SPARQL 1.1 inside SQLite, written in Rust
- Oct 16, 2026Welcome Beam: a 501B open model you can own, customize, and scale
- Oct 15, 2026NVIDIA Nemotron 3.5 Super VL Powers Specialized Agents with Multimodal Reasoning
- Oct 15, 2026System1-Arena: Which Decisions Can Be Left to System 1?