UserSWE: Evaluating Coding Agents in Multi-Turn Conversations with Dynamic Users
We introduce UserSWE for evaluating coding agents' ability to fix repository-level bugs through multi-turn interactions with dynamic users and execution-grounded verdicts at every turn.
ProofPaper ↗
Key points
- As coding agents become more capable, developers increasingly use them to fix bugs through multi-turn interactions.
- Most present a complete issue report in a single turn without interaction.
- We evaluate eleven agents on 72 tasks from 42 repositories and find that agent rankings change with the interaction budget.
- These results indicate that single-turn performance is insufficient to characterize interactive coding ability.
Sources (1)
- [1]UserSWE: Evaluating Coding Agents in Multi-Turn Conversations with Dynamic UsersarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 02:46 PM
We introduce UserSWE for evaluating coding agents' ability to fix repository-level bugs through multi-turn interactions with dynamic users and execution-grounded verdicts at every turn.
As coding agents become more capable, developers increasingly use them to fix bugs through multi-turn interactions.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 21, 2026OpenAI says an AI agent lied about disabling its safety monitoring system
- Oct 21, 2026How Commerce.com and Cloudflare are getting storefronts ready for AI agents
- Oct 21, 2026VeriEvolve: Evaluator-First Evolution for Tool-Using Agents
- Oct 20, 2026Full Guide to OpenAI dots
- Oct 20, 2026Exa 3.0 Search API Tops Perplexity, Brave and Parallel on 8 Benchmarks
- Oct 20, 2026Embodied Task Agents Can Perform Fine Manipulation through Tactile In-Context Learning