Who Sets the Pace of Judge Hacking? Judge Capability versus the Policy-Judge Gap

Key points

• A gold-reward control with the same recipe raised the 1.5B policy's greedy accuracy by +0.054 [0.013, 0.096], so for this policy the flat accuracy is specific to the judge reward; for the 0.5B policy, gold reward also gave no detectable gain.1
• We then ask whether the policy-judge gap or the judge sets how far and how fast RL exploits the judge's false positives.1
• We train 0.5B and 1.5B Qwen2.5 policies against four judges at least as large (Qwen2.5-1.5B/3B/7B, Llama-3.2-3B) and the 1.5B policy against one smaller judge (Qwen3-0.6B).1
• The depth of judge hacking (the plateau) tracks the judge: it correlates with judge AUC (Spearman -0.69 [-0.94, -0.14] over 12 cells), and within a family the larger judge is fooled less deeply.1

Sources (1)

• [1]Who Sets the Pace of Judge Hacking? Judge Capability versus the Policy-Judge Gap
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 11, 05:05 PM

  When a policy is trained with GRPO against a frozen, reference-free LLM judge, the judge accepts more of its answers, but in a post-hoc analysis we detect no gain in held-out greedy accuracy: in all 12 (policy, judge) cells, the seed-mean gain never exceeded 0.04.

  A gold-reward control with the same recipe raised the 1.5B policy's greedy accuracy by +0.054 [0.013, 0.096], so for this policy the flat accuracy is specific to the judge reward; for the 0.5B policy, gold reward also gave no detectable gain.

Extractive summary: sentences quoted from the sources.