Hand2Robot: Object-Centric 4D Hand Reconstruction from Human Demonstrations for Robotics

Key points

• Human demonstrations can capture task-relevant manipulation without using the target robot, but the recorded hand motions cannot directly serve as robot action supervision.1
• Hand reconstruction and motion retargeting provide a direct route to such supervision, yet recovering hand motion alone does not ensure physically consistent hand-object interactions.1
• Given videos of humans performing the target task, Hand2Robot extracts interaction clips, reconstructs object geometry, and generates robot action supervision through hand-object interaction optimization and motion retargeting.1
• Experiments on three real-robot dexterous manipulation tasks further show that reconstructed and retargeted human demonstrations can effectively supplement limited robot data for target-task policy post-training.1

Sources (1)

• [1]Hand2Robot: Object-Centric 4D Hand Reconstruction from Human Demonstrations for Robotics
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 11, 04:48 PM

  To address this issue, we introduce Hand2Robot, an object-centric 4D hand reconstruction framework for robot policy post-training.

  Human demonstrations can capture task-relevant manipulation without using the target robot, but the recorded hand motions cannot directly serve as robot action supervision.

Extractive summary: sentences quoted from the sources.