Vision as Proprioception: Estimating Humanoid Robot Pose from Video

Key points

• To address these issues, we present HumanoidPose, a new framework that treats vision as proprioception to recover full-body joint configurations in world space from video alone.1
• We retarget human motions to sixteen different humanoid embodiments, enabling scalable learning entirely from synthetic data.1
• To improve real-world generalization, we further leverage unlabeled online videos through self-training to develop a 2D robot-pose detector.1
• HumanoidPose enables visual understanding of other robots during interaction and extraction of robot motion from online and generated videos.1

Sources (2)

• [1]Vision as Proprioception: Estimating Humanoid Robot Pose from Video
  Hugging Face Daily Papers · Oct 18, 12:00 AM

  Estimating a humanoid robot's internal state from video has a wide range of real-world applications but remains unexplored.

  To address these issues, we present HumanoidPose, a new framework that treats vision as proprioception to recover full-body joint configurations in world space from video alone.

• [2]Vision as Proprioception: Estimating Humanoid Robot Pose from Video
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 18, 02:10 PM · same content

Extractive summary: sentences quoted from the sources.