LLM Value Functions for Credit Assignment in Agentic RL
Reinforcement learning (RL) for large language model (LLM) agents remains challenging in long-horizon tasks with sparse outcome rewards, where assigning credit to individual actions is difficult.
ProofPaper ↗
Key points
- We introduce LLM Value Function RL (LVRL), which uses the pretrained knowledge and in-context learning ability of an LLM to estimate policy-aware state values without finetuning a separate critic.
- LVRL conditions the value model on group context and calibrates its predictions using known values at the initial and terminal states, enabling step-level credit assignment from estimated value differences.
- We further extend this idea to learning from an expert policy through on-policy distillation (OPD), introducing LLM Value Function OPD (LVOPD), which focuses teacher supervision on actions associated with decreases in estimated value.
- We evaluate our approach across diverse agentic benchmarks: AppWorld, BabyAI, and SWE-bench Verified.
Sources (1)
- [1]LLM Value Functions for Credit Assignment in Agentic RLarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 11, 06:46 PM
Reinforcement learning (RL) for large language model (LLM) agents remains challenging in long-horizon tasks with sparse outcome rewards, where assigning credit to individual actions is difficult.
We introduce LLM Value Function RL (LVRL), which uses the pretrained knowledge and in-context learning ability of an LLM to estimate policy-aware state values without finetuning a separate critic.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 11, 2026ANTE: Anticipate, Then Execute -- Learning Internal World Models for Coding Agents from Verified Beliefs
- Oct 10, 2026VC-OPD: Visual Counterfactual On-Policy Distillation for Grounded Vision-Language Reasoning
- Oct 10, 2026CFG-Policy: Unified Diffusion Policy Learning Framework via Classifier-free Guidance
- Oct 9, 2026Generalized Proximal Policy Optimization: Unifying LLM RL Algorithms from the Perspective of Divergence
- Oct 9, 2026Rethinking Relative Policy Differences Transfer of On-Policy Distillation
- Oct 9, 2026Continually Post-training Small Models via Synthetic Self-distillation
