Flat-RLVR: Sharpness-Aware Reinforcement Learning with Verifiable Rewards for Reasoning Capacity in LLMs

Key points

• While RLVR is highly effective at enhancing the reasoning capabilities of LLMs, it commonly induces distribution sharpening, whereby the policy distribution becomes excessively peaked and rare-yet-correct trajectories are progressively suppressed, thus diminishing sample diversity.1
• To this end, we promote flat-reward optima in RLVR to smooth the policy distribution.1
• We theoretically link flat reward RLVR to reduced distribution sharpening, and empirically show improved sample diversity while maintaining single-attempt accuracy.1
• Our method, called Flat-RLVR, searches for flat reward regions via sharpness-aware minimization during LLM post-training, combining this with GRPO variants to achieve better trade-offs between sampling efficiency and reasoning capacity.1

Sources (1)

• [1]Flat-RLVR: Sharpness-Aware Reinforcement Learning with Verifiable Rewards for Reasoning Capacity in LLMs
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 06:35 AM

  Reinforcement learning with verifiable rewards (RLVR), particularly through group-relative policy optimization (GRPO), has emerged as a central paradigm for the post-training of large language models (LLMs).

  While RLVR is highly effective at enhancing the reasoning capabilities of LLMs, it commonly induces distribution sharpening, whereby the policy distribution becomes excessively peaked and rare-yet-correct trajectories are progressively suppressed, thus diminishing sample diversity.

Extractive summary: sentences quoted from the sources.