Process Reward Modeling with Reliability- and Conservatism-Aware Bayesian Belief

Key points

• However, PRMs may provide inaccurate process rewards for test-time scaling, which we identify as arising from two sources of uncertainty: (1) PRM uncertainty that arises from insufficient supervision data and imperfect PRM training; and (2) policy uncertainty that results from the mismatch between the policy to collect supervision data and the policy for test-time scaling, leading to systematic reward overestimation.1
• To bridge this gap, we propose BayesianPRM, which treats the process reward as a random variable within a reward hypothesis space and maintains a Bayesian belief over it.1
• These two updates yield a reliability- and conservatism-aware Bayesian belief, whose expectation determines the process reward.1
• Experiments on multimodal reasoning benchmarks indicate that BayesianPRM consistently yields more accurate process reward estimates while enhancing test-time scaling performance.1

Sources (1)

• [1]Process Reward Modeling with Reliability- and Conservatism-Aware Bayesian Belief
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 09:54 AM

  Process Reward Models (PRMs) play a central role in test-time scaling for large language model (LLM) reasoning.

  However, PRMs may provide inaccurate process rewards for test-time scaling, which we identify as arising from two sources of uncertainty: (1) PRM uncertainty that arises from insufficient supervision data and imperfect PRM training; and (2) policy uncertainty that results from the mismatch between the policy to collect supervision data and the policy for test-time scaling, leading to systematic re

Extractive summary: sentences quoted from the sources.