Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning

Key points

• Reinforcement learning with verifiable rewards has become a scalable post-training paradigm for large language models.1
• However, training agents over long-horizon interactions presents two challenges: using only terminal rewards makes turn-level credit assignment sparse and high-variance, while sequential mini-batch updates on a fixed rollout batch introduce policy staleness as the current policy drifts from the behavior policy.1
• Treating each interaction turn as a tree node, TreeOPSD branches alternative continuations from shared prefixes and recursively backs up terminal rewards into implicit turn-level credit.1
• Minimizing the divergence between this teacher and the student policy recovers the policy-gradient direction as the tangent at the behavior policy while bounding updates toward a finite target across sequential mini-batches.1

Sources (2)

• [1]Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 09:17 AM

  In this work, we introduce TreeOPSD, a critic-free, tree-aware posterior self-distillation framework for long-horizon agentic reinforcement learning.

  Reinforcement learning with verifiable rewards has become a scalable post-training paradigm for large language models.

• [2]Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning
  Hugging Face Daily Papers · Oct 16, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.