Real2Sem2Real: 3D Semantic Augmentations for Robot Learning from a Single Human RGB Demonstration

Key points

• Human video scales easily and does not need robot hardware, but existing methods require depth sensing, overlay the robot onto 2D video, or generate data only for the demonstrated objects.1
• R2Sem2R tracks the SE(3) pose of each object with DINOv3-distilled Gaussian splats, outperforming existing methods and FoundationPose with estimated metric depth.1
• R2Sem2R maps the human hand to a feasible robot grasp that extends to the cousins.1
• On two real robots and four tasks, we train a vision-language-action policy on data from one human video per task, and policies trained on our data match or exceed policies trained on teleoperation across diverse object instances.1

Sources (1)

• [1]Real2Sem2Real: 3D Semantic Augmentations for Robot Learning from a Single Human RGB Demonstration
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 07:59 PM

  We present Real2Sem2Real (R2Sem2R), a real-to-semantic-to-real pipeline that turns one monocular RGB video of a human demonstration into thousands of robot demonstrations across the semantic object instances of a category.

  Human video scales easily and does not need robot hardware, but existing methods require depth sensing, overlay the robot onto 2D video, or generate data only for the demonstrated objects.

Extractive summary: sentences quoted from the sources.