PTGPO: Prefix Tree Group Policy Optimization
We propose Prefix Tree Group Policy Optimization (PTGPO), a critic-free method that constructs prefix trees over trajectory suffixes within anchor-state groups.
ProofPaper ↗
Key points
- Training language agents on long-horizon interactive tasks is challenging because sparse terminal rewards provide weak credit signals for intermediate actions.
- Existing group-based methods either assign a shared trajectory-level advantage or estimate step-level advantages from state groups, which may conflate the current action with subsequent decisions.
- These prefix-tree credits provide fine-grained learning signals without requiring a critic or additional rollouts.
- Experiments on interactive benchmarks ALFWorld, WebShop and Sokoban show that PTGPO consistently improves over RL baselines, demonstrating its effectiveness.
Sources (1)
- [1]PTGPO: Prefix Tree Group Policy OptimizationarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 10, 08:57 AM
We propose Prefix Tree Group Policy Optimization (PTGPO), a critic-free method that constructs prefix trees over trajectory suffixes within anchor-state groups.
Training language agents on long-horizon interactive tasks is challenging because sparse terminal rewards provide weak credit signals for intermediate actions.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 10, 2026PEARL: Process-Experience Aware Reinforcement Learning for Multi-Turn Agents
- Oct 10, 2026CB-Agent: Concept-Based In-Context Learning for Sequential Action Selection
- Oct 10, 2026SEPM: Self-Evolving Procedural Memory Grounded in Task Graphs for Multi-Agent Collaboration
- Oct 9, 2026Branching On-Policy Distillation