VLM-Guided Contact Retargeting and Generative Partner Modeling for Humanoid-Human Interaction

Key points

• Humanoid-human interaction requires robots to operate under severe partial observability: during close physical contact, the human partner is occluded, truncated, or outside the robot's field of view, causing pose estimation to fail and often necessitating external tracking or privileged human state.1
• First, a VLM-guided contact modeling and retargeting pipeline identifies semantically meaningful contacts and transfers them onto humanoid-human geometry while preserving interaction structure.1
• To address severe partial observability, we introduce generative partner-state modeling objectives that predict partner motion from occluded egocentric views.1
• We evaluate REACT in simulation and on a real Unitree G1 humanoid across five contact-rich interactions.1

Sources (1)

• [1]VLM-Guided Contact Retargeting and Generative Partner Modeling for Humanoid-Human Interaction
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 04:57 PM

  We present REACT, a framework for learning deployable humanoid-human interaction policies from human-human demonstrations.

  Humanoid-human interaction requires robots to operate under severe partial observability: during close physical contact, the human partner is occluded, truncated, or outside the robot's field of view, causing pose estimation to fail and often necessitating external tracking or privileged human state.

Extractive summary: sentences quoted from the sources.