ResearchResearch paperReinforcement Learning · Large Language Models · Safety & Alignment1 source · Oct 10, 2026

Re-executing Coding-Agent Evaluations: Unchanged Decisions and Unresolved Policy Risk

Executable tests grade large language model (LLM) coding-agent patches for leaderboards, rewards and self-improvement.

Two preregistered re-execution studies returned negative results.
SWE-bench Pro v1's offline option rejected none of 59 accepted Go patches whose grading configuration changed.
Re-executing 1,929 selected agent-version/task evaluations from the Darwin Gödel Machine's archive preserved best-agent selection and system-variant ranking, with search history and scores of evaluations not rerun fixed.
We provide an evidence-to-claim summary, lessons for grader audits and an anonymous replication package.

[1]Re-executing Coding-Agent Evaluations: Unchanged Decisions and Unresolved Policy Risk
arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 10, 01:38 AM
Executable tests grade large language model (LLM) coding-agent patches for leaderboards, rewards and self-improvement.
Two preregistered re-execution studies returned negative results.

Extractive summary: sentences quoted from the sources.

Before this