Distill Until Divergence: Error-Truncated Hindsight Distillation for Agentic RL
To address this mismatch, we propose \methodname{}, which routes two complementary hindsight skills according to trajectory outcomes and error boundaries: Positive skills capture successful workflows to reinforce successful trajectories and pre-error decisions, while Negative skills capture failure-specific corrections to target decisive errors.
ProofPaper ↗
Key points
- Long-horizon interaction requires agents to make multiple decisions in a changing environment.
- Outcome-based GRPO lacks fine-grained guidance for intermediate decisions.
- Recent methods distill completed trajectories into reusable skills and use hindsight-conditioned OPD to provide dense token-level credit.
- However, skill-induced probability shifts do not necessarily represent reward-aligned credit; especially after an erroneous action changes subsequent interaction states, skill-conditioned supervision may become mismatched with the current state.
Sources (1)
- [1]Distill Until Divergence: Error-Truncated Hindsight Distillation for Agentic RLarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 07:28 AM
To address this mismatch, we propose \methodname{}, which routes two complementary hindsight skills according to trajectory outcomes and error boundaries: Positive skills capture successful workflows to reinforce successful trajectories and pre-error decisions, while Negative skills capture failure-specific corrections to target decisive errors.
Long-horizon interaction requires agents to make multiple decisions in a changing environment.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 16, 2026Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning
- Oct 15, 2026huggingface/trl v1.16.0
- Oct 15, 2026ReState: Noise-Level-Aware State Correction for Training-Free Object Removal in Few-Step Diffusion Models
- Oct 15, 2026The Density of Thought: Predicting and Steering Reasoning in Large Language Models
- Oct 15, 2026Data World: Scaling Data Environments from Scratch
- Oct 14, 2026RM-OPD: On-Policy Distillation for Generative Reward Models
