ToolOPD: Credit-Aware On-Policy Distillation for Tool-Use Agents

Key points

• However, standard OPD implicitly assumes that credit is uniformly distributed across long-horizon trajectories, assigning the same strength of teacher supervision to different actions.1
• Through credit-distribution analysis and outcome-reversal counterfactual experiments, we find that trajectory credit is concentrated on a small number of critical actions.1
• To address these challenges, we propose ToolOPD, an outcome-aware on-policy distillation framework designed for long-horizon tool use.1
• Extensive experiments across multiple long-horizon tool interaction benchmarks demonstrate that ToolOPD significantly outperforms existing reinforcement learning and self-distillation baselines.1

Sources (1)

• [1]ToolOPD: Credit-Aware On-Policy Distillation for Tool-Use Agents
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 02:41 AM

  On-policy distillation (OPD) enhances the capabilities of small language agents by providing dense, token-level supervision from a teacher model on student-generated trajectories.

  However, standard OPD implicitly assumes that credit is uniformly distributed across long-horizon trajectories, assigning the same strength of teacher supervision to different actions.

Extractive summary: sentences quoted from the sources.