Prorsa: Unleashing the Potential of Prefill-Decode Disaggregation in Agentic Reinforcement Learning

Key points

• Agentic reinforcement learning (RL) is commonly used to enhance the capabilities of large language models (LLMs) in solving complex multi-turn tasks.1
• This paper shows that PD disaggregation can improve rollout efficiency by (a) mitigating interference among data-parallel attention modules, (b) enabling stage-specific parallelism, and (c) reducing end-to-end trajectory completion time.1
• We therefore present Prorsa, an agentic RL framework for prediction-guided prefill-decode resource allocation.1
• Across synchronous and asynchronous agentic RL settings, Prorsa achieves end-to-end rollout speedups of 1.11x-4.05x over prefill-decode colocation.1

Sources (1)

• [1]Prorsa: Unleashing the Potential of Prefill-Decode Disaggregation in Agentic Reinforcement Learning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 11:16 AM

  During rollout, agentic RL exhibits a workload pattern distinct from that of traditional RL: multi-turn interactions repeatedly introduce prefill requests, causing prefill and decode workloads to coexist throughout rollout.

  Agentic reinforcement learning (RL) is commonly used to enhance the capabilities of large language models (LLMs) in solving complex multi-turn tasks.

Extractive summary: sentences quoted from the sources.