Enabling Token-wise Importance Sampling in Off-Policy RLVR with Proximal KL Regularization

Key points

• Token-wise importance weighting avoids multiplying likelihood ratios across a response but generally biases policy gradients.1
• We prove that, under coverage and local expressiveness conditions, full future KL credit makes the exact proximal optimizer the update's unique stationary policy, even for mixtures of older policies, without full-response weights or a fitted critic.1
• We further establish convergence of continuous-time tabular population updates under persistent coverage and time-varying behavior policies, and a finite-sample reward guarantee for linear logistic models under additional assumptions.1
• These results motivate Proximal KL, a practical off-policy RLVR method that separates the proximal reference from the sampling policy and replaces PPO clipping with bounded future KL credit.1

Sources (2)

• [1]Enabling Token-wise Importance Sampling in Off-Policy RLVR with Proximal KL Regularization
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 05:09 PM

  Off-policy data in reinforcement learning with verifiable rewards (RLVR) can arise from asynchronous generation, rollout-training mismatch, and response reuse.

  Token-wise importance weighting avoids multiplying likelihood ratios across a response but generally biases policy gradients.

• [2]Enabling Token-wise Importance Sampling in Off-Policy RLVR with Proximal KL Regularization
  Hugging Face Daily Papers · Oct 13, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.