ReFlow: Replay-Driven Decoupling for Efficient RL Post-Training

Key points

• Existing RL systems commonly rely on asynchronous execution to overlap rollout generation with policy optimization for enhanced efficiency.1
• The dynamic nature of LLM RL and multi-turn agentic workloads further challenges RL system design, as resource demands shift over time, making static resource allocation and generation strategy inefficient.1
• This paper presents ReFlow, a system-algorithm co-designed RL framework for efficient LLM post-training.1
• Experiments across representative single-turn reasoning and multi-turn agentic RL tasks show that ReFlow improves training throughput by up to $2.06\times$ over state-of-the-art synchronous and asynchronous RL systems, while maintaining comparable or better model performance.1

Sources (1)

• [1]ReFlow: Replay-Driven Decoupling for Efficient RL Post-Training
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 10:21 AM

  Reinforcement learning (RL) has been a critical paradigm for improving large language model (LLM) reasoning and agentic capabilities, but its efficiency remains a major bottleneck.

  Existing RL systems commonly rely on asynchronous execution to overlap rollout generation with policy optimization for enhanced efficiency.

Extractive summary: sentences quoted from the sources.