Explorer-Zero: Improving Self-Play for Long-Horizon Tasks with Decomposable Verifiable Rewards

Key points

• Training LLM agents on long-horizon interactive tasks with reinforcement learning typically relies on human-annotated tasks and reliable verification, both expensive to construct for stateful environments.1
• Self-play offers an alternative, but must generate achievable, compositional tasks and provide feedback beyond binary outcomes.1
• We show theoretically that subtask rewards reduce collapse among unsuccessful rollouts when they complete different numbers of subtasks.1
• Without human-annotated training tasks, Explorer-Zero outperforms the evaluated self-play baselines across ALFWorld, ScienceWorld, and AppWorld; on AppWorld, our 30B agent reaches 59.67% task success, approaching 62.65% for RL on human-annotated tasks.1

Sources (1)

• [1]Explorer-Zero: Improving Self-Play for Long-Horizon Tasks with Decomposable Verifiable Rewards
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 09:29 AM

  We propose Explorer-Zero, a self-play framework that jointly generates tasks and fine-grained, verifiable rewards through environment interaction.

  Training LLM agents on long-horizon interactive tasks with reinforcement learning typically relies on human-annotated tasks and reliable verification, both expensive to construct for stateful environments.

Extractive summary: sentences quoted from the sources.