Gibbs-Gap Policy Optimization for Off-Policy RL with Verifiable Rewards

Key points

• Reinforcement learning with verifiable rewards often trains language models on responses generated by a different, possibly stale behavior policy.1
• Correcting this mismatch with learner-to-behavior importance weights can introduce high variance for long responses.1
• We derive GGPO from KL-regularized policy optimization: on policy, its population loss is exactly the KL-regularized optimality gap, and under suitable coverage the learner's single-response expected reward is at least the behavior policy's best-of-$K$ reward, minus terms for the raw finite-group loss, regularization, and finite training data.1
• We characterize reference refresh through exact Gibbs-target composition and a finite-data final-policy reward bound under local reference control and tokenwise behavior mismatch.1

Sources (1)

• [1]Gibbs-Gap Policy Optimization for Off-Policy RL with Verifiable Rewards
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 06:59 PM

  We introduce Gibbs-Gap Policy Optimization (GGPO), which minimizes a log-mean-exp contrast of KL-adjusted rewards among responses to the same prompt, without learner-to-behavior importance weights.

  Reinforcement learning with verifiable rewards often trains language models on responses generated by a different, possibly stale behavior policy.

Extractive summary: sentences quoted from the sources.