RL-ARC: Calibrating Large Reasoning Models via Reasoning-guided Uncertainty

Key points

• Language models (LMs) are commonly trained with Reinforcement Learning with Verifiable Rewards (RLVR) to enhance their reasoning capabilities.1
• However, since RLVR does not explicitly account for calibration during training, it can lead to severe calibration degradation, including overconfidence.1
• Recent calibration-aware training methods for LMs, which incorporate objectives for uncertainty estimation into training, improve calibration but still exhibit overconfidence under distribution shift, while sacrificing reasoning performance.1
• Specifically, RL-ARC leverages reasoning confidence as an auxiliary signal for calibrating answer confidence, applying it as reasoning-guided regularization for correct cases and as an overconfidence penalty for incorrect cases.1

Sources (1)

• [1]RL-ARC: Calibrating Large Reasoning Models via Reasoning-guided Uncertainty
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 06:45 AM

  To this end, we propose RL-ARC, a calibration-aware training framework that jointly leverages reasoning confidence and answer confidence.

  Language models (LMs) are commonly trained with Reinforcement Learning with Verifiable Rewards (RLVR) to enhance their reasoning capabilities.

Extractive summary: sentences quoted from the sources.