Forging More Tasks into Better Harnesses: Agentic Harness Evolution via Successive Scale Racing

Key points

• While more evolution tasks should intuitively improve performance, our preliminary study reveals a paradoxical failure mode: the performance of existing methods degrades as the task pool expands.1
• Our further analysis demonstrates that this degradation stems from suboptimal evidence selection during optimization, as different evidence combinations lead to substantially different evolution outcomes.1
• To address this issue, we propose Successive Scale Racing (SSR), a framework that adaptively searches for optimal evidence combinations in each evolution round.1
• SSR constructs diverse evidence combinations based on the corresponding task quality and coverage, and efficiently races the resulting candidate harnesses under a small compute budget.1

Sources (1)

• [1]Forging More Tasks into Better Harnesses: Agentic Harness Evolution via Successive Scale Racing
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 06:36 PM

  Agentic harness evolution optimizes an agent's instructions, tools, and control flow by using its task execution experiences as evolving evidence.

  While more evolution tasks should intuitively improve performance, our preliminary study reveals a paradoxical failure mode: the performance of existing methods degrades as the task pool expands.

Extractive summary: sentences quoted from the sources.