Gen2Dex: Learning Dexterous Manipulation from Simulation-Initialized Generated Video Demonstrations

Key points

• We study a setting in which a fully specified simulated scene is used to generate videos that specify the demonstrated manipulation behavior.1
• We present an end-to-end framework for converting such simulation-grounded videos into physically executable dexterous robot demonstrations.1
• The recovered human motion is then retargeted and physically grounded using residual reinforcement learning, where a backward curriculum and a cache of physically valid reset states enable the policy to correct imperfect video-derived references while preserving the demonstrated manipulation strategy.1
• We further demonstrate diverse rigid and articulated manipulation tasks in simulation and successful open-loop transfer of the optimized trajectories to a real dexterous robot.1

Sources (1)

• [1]Gen2Dex: Learning Dexterous Manipulation from Simulation-Initialized Generated Video Demonstrations
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 17, 02:50 AM

  Human video demonstrations provide rich manipulation strategies for dexterous manipulation learning.

  We study a setting in which a fully specified simulated scene is used to generate videos that specify the demonstrated manipulation behavior.

Extractive summary: sentences quoted from the sources.