ACCRUE: Continual Off-Policy Reinforcement Learning for Robot Foundation Models

Key points

• Reinforcement learning (RL) post-training improves robot foundation models (RFMs) beyond their demonstrations, but real-world deployment demands more: learning new tasks efficiently while retaining previously acquired behaviors.1
• Existing continual RL methods on top of VLAs mitigate forgetting but rely on on-policy updates with limited forward transfer, whereas off-policy methods improve single-task sample efficiency without addressing learner persistence across tasks.1
• The inherited critic adapts only on current-task transitions and guides current-task actor learning, while prior behavior is preserved by distilling the actor toward cached targets on a small number of retained episodes.1
• Across contact-rich assembly sequences in simulation and on real-robot hardware, ACCRUE accelerates subsequent-task learning with near-zero forgetting and significantly reduces real-robot training time by 31.0% to 68.5%.1

Sources (2)

• [1]ACCRUE: Continual Off-Policy Reinforcement Learning for Robot Foundation Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 03:53 PM

  We propose ACCRUE, a continual off-policy RL framework that carries lightweight actor and critic networks across tasks, allowing each stage to reuse behavioral and value knowledge acquired through prior RL.

  Reinforcement learning (RL) post-training improves robot foundation models (RFMs) beyond their demonstrations, but real-world deployment demands more: learning new tasks efficiently while retaining previously acquired behaviors.

• [2]ACCRUE: Continual Off-Policy Reinforcement Learning for Robot Foundation Models
  Hugging Face Daily Papers · Oct 13, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.