C3: Rethinking Process Supervision via Confidence-Guided Credit Correction in Multi-Turn Reinforcement Learning

Key points

• Large language models are increasingly trained as multi-turn agents that interact with external tools.1
• However, outcome-only reinforcement learning can produce a form of misguided reinforcement: even on successful trajectories, undesirable local behaviors may receive positive token-level advantages and be favored by the policy update.1
• Across multiple multi-turn benchmarks, C$^3$ improves performance under both {critic-based and critic-free reinforcement learning methods}.1
• Our results show that local credit correction provides an effective form of process supervision for outcome-driven RL.1

Sources (1)

• [1]C3: Rethinking Process Supervision via Confidence-Guided Credit Correction in Multi-Turn Reinforcement Learning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 07:18 AM

  To address this issue, we propose Confidence-Guided Credit Correction (C$^3$), a plug-in method for local credit correction.

  Large language models are increasingly trained as multi-turn agents that interact with external tools.

Extractive summary: sentences quoted from the sources.