Double-Rollout Reinforcement Learning with Sampling-based Reward Estimation

Key points

• We present empirical evidence for this mismatch and hypothesize that the resulting reward noise contributes to instability in reinforcement learning for both reasoning accuracy and token efficiency.1
• To address this credit assignment challenge, this paper proposes double-rollout reinforcement learning.1
• For each sampled rollout, we truncate its reasoning trace, sample multiple continuations from the truncated prefix, and replace the original reward with the mean continuation reward, yielding a more robust estimate of reasoning quality.1
• These findings suggest that the double-rollout reward estimation is an effective and cost-efficient way to improve both the accuracy and token efficiency of reasoning RL.1

Sources (1)

• [1]Double-Rollout Reinforcement Learning with Sampling-based Reward Estimation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 11:16 AM

  Reinforcement learning with verifiable rewards (RLVR) implemented with Group Relative Policy Optimization (GRPO) has driven substantial progress in training reasoning language models.

  We present empirical evidence for this mismatch and hypothesize that the resulting reward noise contributes to instability in reinforcement learning for both reasoning accuracy and token efficiency.

Extractive summary: sentences quoted from the sources.