DeltaTTG: Training Multi-Turn Agents with a Token-to-Go Progress Meter
We introduce DeltaTTG to learn a reusable progress signal through a shared progress meter, using low-cost supervision from diverse policies' recorded trajectories without task-specific progress rules.
ProofPaper ↗
Key points
- Reinforcement learning with verifiable rewards (RLVR) for multi-turn agents relies on sparse terminal rewards that say little about individual steps.
- Within-group step comparisons depend on reliable state matching and sufficiently diverse sampled outcomes, while direct step scoring may require costly external evaluators, human annotations, or task-specific heuristics.
- During actor training, changes in the frozen meter's estimates redistribute outcome-derived advantages across steps without altering the reward, preserving each trajectory's advantage sign and token-average scale.
- On ALFWorld, PaperSearchQA, and BFCLv3, DeltaTTG improves the average score over GRPO by 5.7 and 7.2 points with Qwen3-4B and Llama-3.1-8B-Instruct, respectively, outperforms all compared RL baselines on average, and keeps actor-training GPU-hours close to GRPO.
Sources (2)
- [1]DeltaTTG: Training Multi-Turn Agents with a Token-to-Go Progress MeterHugging Face Daily Papers · Oct 13, 12:00 AM
We introduce DeltaTTG to learn a reusable progress signal through a shared progress meter, using low-cost supervision from diverse policies' recorded trajectories without task-specific progress rules.
Reinforcement learning with verifiable rewards (RLVR) for multi-turn agents relies on sparse terminal rewards that say little about individual steps.
- [2]DeltaTTG: Training Multi-Turn Agents with a Token-to-Go Progress MeterarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 03:49 AM · same content
Extractive summary: sentences quoted from the sources.
Before this
- Oct 12, 2026Reinforcement Learning In Partially Verifiable Environments for Colon Cancer Treatment Planning
- Oct 12, 2026SFT Leaves a Reproducible Signature in Activation Geometry: Calibrating Post-Training Interpretability Against Run-to-Run Noise
- Oct 12, 2026TPPO: Tracking-Value PPO For Language Model Reasoning
- Oct 11, 2026SpecFit: Not All Draft Tokens Cost the Same in Offloaded MoE Inference
- Oct 11, 2026RSIPO: Recursive Self-Improving Policy Optimization for Agentic Reinforcement Learning via Skill-Policy Co-Evolution
- Oct 10, 2026Predicting how model propensities generalize out of distribution
