Preserving Reasoning Diversity in Reinforcement Learning with Verifiable Rewards via Adaptive Replay

Key points

• We study this phenomenon from the perspective of finite-sample learning dynamics and show that limited on-policy rollouts can progressively suppress semantically distinct but equally rewarded reasoning modes.1
• Motivated by this observation, we propose TAR, a Targeted Adaptive Replay framework that preserves diverse correct reasoning behaviors already present in the base policy.1
• TAR constructs a compact memory bank of diverse base-model solutions and selectively replays them through an adaptive mechanism that activates only when stored behaviors are being suppressed during RLVR.1
• Across six mathematical reasoning benchmarks and two Qwen2.5 model scales, TAR consistently achieves a better balance between reasoning performance and diversity preservation than existing diversity-preserving post-training methods, while substantially reducing the forgetting of correct base-model solutions.1

Sources (1)

• [1]Preserving Reasoning Diversity in Reinforcement Learning with Verifiable Rewards via Adaptive Replay
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 09:04 PM

  Reinforcement Learning with Verifiable Rewards (RLVR) improves the reasoning capabilities of large language models, but often reduces the diversity of their reasoning behaviors.

  We study this phenomenon from the perspective of finite-sample learning dynamics and show that limited on-policy rollouts can progressively suppress semantically distinct but equally rewarded reasoning modes.

Extractive summary: sentences quoted from the sources.