TrajEvolve: Trajectory Evolution for Reinforcement Learning with Hindsight Credit Assignment

Key points

• The Reinforcement Learning (RL) paradigm has achieved remarkable success in enhancing the reasoning capabilities of Large Language Models (LLMs).1
• However, when applied to long-horizon tasks, it faces severe challenges: i) Due to the high task complexity, the model struggles to sample enough successful trajectories, resulting in slow convergence. ii) Relying solely on sparse final rewards causes all steps to be updated indiscriminately, encouraging redundant step generation.1
• This dilemma stems from the limitations: existing RL paradigms rely entirely on the reasoning capability of base models, lacking the ability to actively optimize trajectories.1
• TrajEvolve not only relies on the trajectory inferred by the model but also generates "evolutionary trajectories" by refining low-quality steps in historical trajectories, thereby exposing the policy to high-quality compositions that lie within the base model's support but are rarely sampled within practical training budgets.1

Sources (1)

• [1]TrajEvolve: Trajectory Evolution for Reinforcement Learning with Hindsight Credit Assignment
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 11:48 AM

  To address the above issues, we propose a novel trajectory evolution paradigm with hindsight credit assignment, termed TrajEvolve.

  The Reinforcement Learning (RL) paradigm has achieved remarkable success in enhancing the reasoning capabilities of Large Language Models (LLMs).

Extractive summary: sentences quoted from the sources.