Two Clocks: Why Larger Rollout Batches Learn Slower but Fail on Schedule in Reinforcement Learning with Verifiable Rewards
Reinforcement learning with verifiable rewards (RLVR) relies on costly model-generated responses, motivating larger rollout batches and square-root learning-rate scaling.
ProofPaper ↗
Key points
- We develop a two-clock framework that separates the samples needed to make progress from those consumed before collapse.
- Under a local quadratic model, the square-root rule preserves first-order drift per sample but not progress: larger batches take fewer, longer updates and lose more to curvature, yielding a learning-rate-dependent critical batch beyond which sample efficiency degrades.
- Controlled GRPO experiments on Qwen2.5-Math-1.5B across a $16\times$ batch range support both predictions.
- Finally, we use the two-clock framework to derive a simple multi-step RLVR schedule: start below the critical batch and switch to a larger batch before failure.
Sources (1)
- [1]Two Clocks: Why Larger Rollout Batches Learn Slower but Fail on Schedule in Reinforcement Learning with Verifiable RewardsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 05:58 PM
Reinforcement learning with verifiable rewards (RLVR) relies on costly model-generated responses, motivating larger rollout batches and square-root learning-rate scaling.
We develop a two-clock framework that separates the samples needed to make progress from those consumed before collapse.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 13, 2026Run fast, private agents on-device with LFM2.5-24B-A2B
- Oct 13, 2026ProAS: A Progress-Aware Advantage Shaping Framework for Multi-Turn Agentic Reinforcement Learning
- Oct 13, 2026Gemini 4 Argon: Built to solve your hardest problems
- Oct 13, 2026DeltaTTG: Training Multi-Turn Agents with a Token-to-Go Progress Meter
- Oct 12, 2026VAttnPO: Visual Attention-Guided Policy Optimization
- Oct 12, 2026Double-Rollout Reinforcement Learning with Sampling-based Reward Estimation