When Does Momentum Help in Reinforcement Learning with Verifiable Rewards?

Key points

• Momentum is a classical tool for accelerating convex optimization and a standard component of modern large language model training.1
• We study this question through a bandit formulation of RLVR with bias-corrected exponential moving average (EMA) momentum, where the EMA coefficient $β$ controls how quickly the weights on past gradients decay.1
• When policy gradients are evaluated exactly, we prove that increasing $β$ within a range that prevents overshoot strictly accelerates optimization.1
• We show that as the learning rate $η$ decreases, the expected gain with fixed $β$ vanishes faster than these fluctuations, whereas increasing $β$ to keep $η/(1-β)$ within the safe regime preserves a nonvanishing gain that eventually dominates them.1

Sources (1)

• [1]When Does Momentum Help in Reinforcement Learning with Verifiable Rewards?
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 04:11 PM

  Yet recent empirical work has questioned the usefulness of first-moment momentum in reinforcement learning with verifiable rewards (RLVR).

  Momentum is a classical tool for accelerating convex optimization and a standard component of modern large language model training.

Extractive summary: sentences quoted from the sources.