RM-OPD: On-Policy Distillation for Generative Reward Models

Key points

• However, Mix-RL suffers from two critical bottlenecks: the reward sparsity induced by outcome rewards, and the cross-domain interference arising from jointly optimizing heterogeneous capabilities, which together give rise to imbalanced capability acquisition and pervasive reward hacking.1
• Inspired by the success of On-Policy Distillation (OPD) in multi-domain capability integration, we introduce RM-OPD (Reward Modeling via On-Policy Distillation), a two-stage framework that separates specialist acquisition from capability integration.1
• Across multiple RM benchmarks and two base models, RM-OPD consistently outperforms vanilla Mix-RL and other strong baselines, achieving overall superior performance and exhibiting an emergent "teacher-surpassing" effect.1
• These results establish RM-OPD as a promising paradigm for building generalist GRMs.1

Sources (1)

• [1]RM-OPD: On-Policy Distillation for Generative Reward Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 01:20 PM

  Reinforcement Learning on Mixed-domain preference data (Mix-RL) has become the mainstream approach for training general-purpose Generative Reward Models (GRMs).

  However, Mix-RL suffers from two critical bottlenecks: the reward sparsity induced by outcome rewards, and the cross-domain interference arising from jointly optimizing heterogeneous capabilities, which together give rise to imbalanced capability acquisition and pervasive reward hacking.

Extractive summary: sentences quoted from the sources.