SWE-DIVE: Multi-Turn Coding Agent Evaluation with Dynamic User Simulation
To benchmark these abilities, we developed SWE-DIVE, a dynamic, interactive validation and evaluation framework for benchmarking coding agents in a multi-turn environment.
ProofPaper ↗
Key points
- Coding agents are typically evaluated using one-shot, well-formulated prompts.
- Benchmarks such as SWE-bench use existing GitHub issues as input for a coding agent and evaluate whether the agent successfully solves the task.
- To achieve this, we introduce a user simulator that can simulate different behavioral patterns and communicate with the coding agent.
- We also introduce a conversation validator that evaluates the user simulator's outputs to determine whether they fulfill the initial requirements and guide the agent toward successfully solving the task.
Sources (1)
- [1]SWE-DIVE: Multi-Turn Coding Agent Evaluation with Dynamic User SimulationarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 05:06 PM
To benchmark these abilities, we developed SWE-DIVE, a dynamic, interactive validation and evaluation framework for benchmarking coding agents in a multi-turn environment.
Coding agents are typically evaluated using one-shot, well-formulated prompts.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 21, 2026ollama/ollama v0.40.8
- Oct 20, 2026How Eon built multi-cloud investigation with an MCP Gateway on Amazon Bedrock AgentCore
- Oct 20, 2026Do Diffusion LLMs Scale? Testing Mercury 3 + Decide
- Oct 20, 2026Cache Engineering for Agents
- Oct 20, 2026Evolving Agent Interfaces
- Oct 20, 2026Automating ML for Science: Can Frontier Agents Climb Scientific Hills in the Wild?