TPPO: Tracking-Value PPO For Language Model Reasoning

Key points

• While value evolution along trajectory characterizes relative token-level credit, yet standard critics often fail to track it under sparse terminal rewards.1
• To model this evolution, we analyze the relation between successive prefix values and show that they admit a Bayesian decomposition with additive updates in log-odds space.1
• We use this additive structure to construct Tracking-Value PPO (\method), a cumulative value parameterization that sums position-wise scalar outputs into value logits.1
• These outputs are learned through terminal-outcome supervision on the reconstructed values. \method improves aggregate mathematical reasoning scores over PPO by 7.0 and 8.4 points on Qwen3-4B-Base and Qwen3-8B-Base, respectively, while improving out-of-distribution generalization.1

Sources (2)

• [1]TPPO: Tracking-Value PPO For Language Model Reasoning
  Hugging Face Daily Papers · Oct 12, 12:00 AM

  Proximal Policy Optimization (PPO) is widely used in reinforcement learning with verifiable rewards for language-model reasoning.

  While value evolution along trajectory characterizes relative token-level credit, yet standard critics often fail to track it under sparse terminal rewards.

• [2]TPPO: Tracking-Value PPO For Language Model Reasoning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 01:47 PM · same content

Extractive summary: sentences quoted from the sources.