Spectral-Driven Adaptive-Rank Optimizer for Large Language Model Reinforcement Learning with Verifiable Rewards

Key points

• This raises a fundamental question: does RLVR induce a distinct optimization geometry that calls for a different optimizer?1
• We investigate this question through the spectral structure of LLM gradients and find that gradient energy progressively concentrates into a shrinking effective subspace, which is characteristic of RLVR, as it is absent under supervised fine-tuning (SFT) and is largely removed when the verifiable reward is replaced by a random one.1
• These findings motivate a new optimization principle: rather than prescribing the update rank externally, an optimizer should adapt its rank to the evolving spectral structure of the gradient.1
• We introduce Ares (Adaptive Rank-Effective Spectral Optimization), which estimates the effective update rank from the gradient's spectral energy distribution, tracks its evolution, and confines the gradient signal driving each update to the resulting low-rank subspace.1

Sources (2)

• [1]Spectral-Driven Adaptive-Rank Optimizer for Large Language Model Reinforcement Learning with Verifiable Rewards
  Hugging Face Daily Papers · Oct 12, 12:00 AM

  Reinforcement learning with verifiable rewards (RLVR) is reshaping how large language models (LLMs) acquire reasoning capabilities, yet the optimizers used remain largely inherited from teacher-forcing training.

  This raises a fundamental question: does RLVR induce a distinct optimization geometry that calls for a different optimizer?

• [2]Spectral-Driven Adaptive-Rank Optimizer for Large Language Model Reinforcement Learning with Verifiable Rewards
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 10:21 AM · same content

Extractive summary: sentences quoted from the sources.