SWEeper-Bench: Can Agents Discover Bugs in Interactive Software?
Coding agents increasingly interact with users by generating interactive software on the fly rather than replying in text, yet many bugs in such software reveal themselves only after multi-step interactions.
ProofPaper ↗
Key points
- Today, users must discover these bugs and wait for fixes, making the development loop slow and expensive.
- We introduce SWEeper-Bench, 200 tasks built from real bug fixes in 200 widely used open-source web applications.
- Through stage-by-stage analysis of agent trajectories, we show that the biggest bottleneck is identifying the buggy behavior: agents often reach the relevant feature but never trigger the failure or fail to recognize it.
- SWEeper-Bench measures progress toward agents that close this loop themselves, delivering software that works without waiting for user feedback.
Sources (1)
- [1]SWEeper-Bench: Can Agents Discover Bugs in Interactive Software?Hugging Face Daily Papers · Oct 8, 12:00 AM
Coding agents increasingly interact with users by generating interactive software on the fly rather than replying in text, yet many bugs in such software reveal themselves only after multi-step interactions.
Today, users must discover these bugs and wait for fixes, making the development loop slow and expensive.
Extractive summary: sentences quoted from the sources.