MultiAgent-OPD: End-to-End Training of Hierarchical Multi-Agent Systems with On-Policy Distillation
We present MultiAgent-OPD, which combines on-policy distillation with reinforcement learning to jointly train a shared-parameter Leader-Searcher-Answerer team end to end.
ProofPaper ↗
Key points
- In a hierarchical multi-agent system, a Leader assigns tasks and subagents carry them out.
- Jointly training the Leader and the subagents is difficult because a final answer gives only indirect feedback to the subagents.
- We first train a Teacher in a Searcher-Answerer workflow using answer correctness.
- Across seven datasets, the trained system reaches 38.68% macro avg@4, exceeding Prompted Critic by 4.06 percentage points and C-MAS + RL, the trained cascaded multiagent system, by 2.61 percentage points.
Sources (1)
- [1]MultiAgent-OPD: End-to-End Training of Hierarchical Multi-Agent Systems with On-Policy DistillationarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 10:01 AM
We present MultiAgent-OPD, which combines on-policy distillation with reinforcement learning to jointly train a shared-parameter Leader-Searcher-Answerer team end to end.
In a hierarchical multi-agent system, a Leader assigns tasks and subagents carry them out.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 16, 2026Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning
- Oct 15, 2026AlphaBet: Reinforcement Learning for LLM Event Forecasting Agents
- Oct 15, 2026Learning to Steer LLM Agents and Multi-Agent Systems
- Oct 15, 2026[AINews] Modal’s $1B ARR, $16B Series D makes it NYC’s newest AI Infra Decacorn, 3x usage and 3x valuation in 5 months, 15x in 1 yr
- Oct 15, 2026Skill-Decoupled On-Policy Expert Distillation for Unified Vision-Language Models
- Oct 15, 2026Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation
