ResearchResearch paperReinforcement Learning1 source · Oct 12, 2026

Gradient Routing Localizes and Removes Unintended Behaviors in RL

We propose a mitigation to reward hacking in reinforcement learning.

Our setup involves a misspecified reward function that gives rise to unintended behavior, and a monitor used to detect unintended behavior.
To address this problem, we introduce GRAFT: Gradient Routing Adapter Fine Tuning.
We demonstrate these benefits in RL environments where reward penalties cause models to learn to reward hack in undetected ways, training models ranging from 135M to 8B parameters.
Additionally, we show a similar effect at a larger scale by distilling frontier LLM trajectories containing reward hacking into Qwen3-32B, without teaching that model to reward hack.

[1]Gradient Routing Localizes and Removes Unintended Behaviors in RL
arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 04:28 PM
We propose a mitigation to reward hacking in reinforcement learning.
Our setup involves a misspecified reward function that gives rise to unintended behavior, and a monitor used to detect unintended behavior.

Extractive summary: sentences quoted from the sources.

Before this