Anatomy of Off-Policy Policy Gradient: Importance Sampling, KL Regularization, and Baselines

Key points

• We study when such off-policy deployment is justified, and what roles its core ingredients, including importance sampling, KL regularization, and baselines, play in handling the resulting distribution shift.1
• Focusing on the KL route, we identify it with the group-relative policy gradient (GRPG) update, establish an equivalence to a trajectory-level Bellman residual minimization objective, and obtain a finite-sample regret bound under all-policy coverage.1
• We further uncover a hidden role of baselines in the off-policy setting: a constant shift in the baseline implements pessimism, the standard remedy in offline RL for partial data coverage.1
• Together, these results offer a unified theoretical view of off-policy policy gradient in LLM post-training and connect it to classical ideas in offline RL.1

Sources (1)

• [1]Anatomy of Off-Policy Policy Gradient: Importance Sampling, KL Regularization, and Baselines
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 03:27 AM

  Reinforcement learning has become a central tool for large language model (LLM) post-training, where policy gradient methods are routinely deployed off-policy, even though vanilla policy gradient assumes on-policy sampling.

  We study when such off-policy deployment is justified, and what roles its core ingredients, including importance sampling, KL regularization, and baselines, play in handling the resulting distribution shift.

Extractive summary: sentences quoted from the sources.