SWE-Residual: Evaluating Starting-State Robustness of Coding Agents

Key points

• Software engineering (SWE) benchmarks are the primary way to measure the capabilities of coding agents: an agent is placed in a code repository or machine environment, given a request, and judged by hidden tests.1
• These benchmarks almost always start the agent from a clean state that contains no earlier work on the task.1
• SWE-Residual contains 198 Residual States drawn from 111 tasks in DeepSWE and Terminal-Bench, where the earlier work modifies a code repository or a whole machine environment.1
• Failed continuations show a pattern we call acceptance bias: agents treat the inherited work as the definition of what remains to be done instead of checking it against the request.1

Sources (1)

• [1]SWE-Residual: Evaluating Starting-State Robustness of Coding Agents
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 07:00 AM

  We introduce SWE-Residual, a benchmark for Starting-State Robustness: the ability to maintain performance on a task when the starting state already contains earlier work on that task.

  Software engineering (SWE) benchmarks are the primary way to measure the capabilities of coding agents: an agent is placed in a code repository or machine environment, given a request, and judged by hidden tests.

Extractive summary: sentences quoted from the sources.