Diagnosing and Correcting Reward Hacking in Verifiable Instruction Following

Key points

• Verifiable instruction-following benchmarks emphasize output-constraint satisfaction but can overlook whether responses meaningfully answer the underlying request, creating an evaluation blind spot.1
• We show that frontier models can receive full credit for responses with little substantive answer content.1
• To address this reward hacking, we introduce CALIF, a lightweight method that augments verifier rewards with a calibrated likelihood-based bonus to preserve response quality.1
• Across models ranging from 1.7B to 8B in reasoning and non-reasoning modes, CALIF improves constraint satisfaction on IFBench while maintaining response quality near or above pre-RL levels.1

Sources (2)

• [1]Diagnosing and Correcting Reward Hacking in Verifiable Instruction Following
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 03:42 AM

  Using the same verifier-based criterion as a training reward can induce reward hacking, as publicly released checkpoints exhibit higher constraint satisfaction but lower response quality after reinforcement learning with verifiable rewards.

  Verifiable instruction-following benchmarks emphasize output-constraint satisfaction but can overlook whether responses meaningfully answer the underlying request, creating an evaluation blind spot.

• [2]Diagnosing and Correcting Reward Hacking in Verifiable Instruction Following
  Hugging Face Daily Papers · Oct 14, 03:42 AM · same content

Extractive summary: sentences quoted from the sources.