Trajectory-Aware Policy Optimization: Closing the State Distribution Gap in Long-Horizon LLM RL

Key points

• Reinforcement learning (RL) has become a dominant paradigm for enhancing LLMs' reasoning capabilities.1
• Yet long-horizon autoregressive rollouts introduce a fundamental limitation: local policy mismatch can compound into severe state distribution shift over trajectories.1
• Existing LLM RL methods, such as GRPO and GSPO, control policy mismatch at the token or sequence level, but overlook this trajectory-induced shift, causing training instability and eventual collapse as the horizon grows.1
• Extensive experiments demonstrate that TAPO significantly surpasses existing LLM RL algorithms across eight competition-level benchmarks, achieving up to 48% and 12% improvement over GRPO on mathematical reasoning and algorithmic coding, with substantially enhanced training stability.1

Sources (1)

• [1]Trajectory-Aware Policy Optimization: Closing the State Distribution Gap in Long-Horizon LLM RL
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 05:22 AM

  To close the state distribution gap, we propose Trajectory-Aware Policy Optimization (TAPO), a principled framework that theoretically derives the state-distribution correction from the off-policy policy gradient and incorporates it into the RL objective.

  Reinforcement learning (RL) has become a dominant paradigm for enhancing LLMs' reasoning capabilities.

Extractive summary: sentences quoted from the sources.