ProbePO: Sparse Probes, Redistributed Credit for Critic-Free LLM Reinforcement Learning

Key points

• We cast the missing middle as budgeted measurement: with a limited rollout budget, which partial solutions should be evaluated, and how should their value estimates refine credit?1
• ProbePO selects partial-solution states with a reward-free uncertainty score and estimates their expected verifier rewards under a capped continuation budget.1
• ProbePO redistributes, rather than replaces, GRPO's trajectory-level advantage across tokens while preserving each trajectory's mean token-level advantage; probe continuations serve only as measurements.1
• Against dense-probing VinePPO on DeepSeekMath-7B, ProbePO achieves higher accuracy on all six benchmarks with 76% less end-to-end training time.1

Sources (1)

• [1]ProbePO: Sparse Probes, Redistributed Credit for Critic-Free LLM Reinforcement Learning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 17, 07:19 AM

  Reinforcement learning from verifiable rewards assigns one terminal reward to an entire reasoning trajectory, obscuring how individual reasoning steps contributed to the outcome.

  We cast the missing middle as budgeted measurement: with a limited rollout budget, which partial solutions should be evaluated, and how should their value estimates refine credit?

Extractive summary: sentences quoted from the sources.