Mitigating Reward Hacking in Preference Reward Models with Evolving Penalties
Reinforcement learning for language models has moved from preference reward models to verifiable rewards, which sustain much longer optimization.
ProofPaper ↗
Key points
- Preference models are prone to reward hacking: the policy raises their score without improving response quality, or even while degrading it.
- We introduce Dynalize, which mitigates reward hacking online, during policy training.
- For example, we find that our method suppresses patterns incentivized by preference models that, without penalties, spread to most responses, and even to the model's reasoning, which the reward model does not explicitly score.
- Furthermore, we show that its gains hold in multi-turn training, and even after we retrain the reward model on the discovered behaviors.
Sources (1)
- [1]Mitigating Reward Hacking in Preference Reward Models with Evolving PenaltiesarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 02:34 PM
Reinforcement learning for language models has moved from preference reward models to verifiable rewards, which sustain much longer optimization.
Preference models are prone to reward hacking: the policy raises their score without improving response quality, or even while degrading it.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 19, 2026FrameGRPO: Incentivizing Video Reasoning via Multi-Frame Rollouts and Temporal Credit
- Oct 19, 2026Adaptive Decoding Methods for Token-Efficient Reinforcement Learning
- Oct 18, 2026Conformal Prediction-Enabled Sequence-level Policy Optimization for Reasoning Language Models
- Oct 17, 2026ProbePO: Sparse Probes, Redistributed Credit for Critic-Free LLM Reinforcement Learning
- Oct 17, 2026Axiomatic World Modeling for Physics Reasoning
- Oct 16, 2026Training-Free Prompt Selection for Efficient Reinforcement Learning With Partial Rollouts