MoCaP: Mobile Code-as-Policies for Zero-Shot Mobile Manipulation from Passive Human Videos

Key points

• Passive human videos of everyday activity offer robots access to a massive amount of human experience without requiring a demonstration for each task.1
• Guided by passive human experience and current observations, MoCaP uses a closed-loop code-as-policy to coordinate planning and tool use primitives to solve motions under the robot's geometry and kinematic constraints.1
• On a real mobile manipulator, MoCaP achieves 80% success on each of two mobile manipulation tasks in an indoor environment whereas existing approaches cannot succeed at the task.1
• Video-derived spatial memory improves object-navigation success from 70% to 100%, and human approach prior raises the rate of finding a reachable base pose from 0% to 100% at an unfavorable start location.1

Sources (1)

• [1]MoCaP: Mobile Code-as-Policies for Zero-Shot Mobile Manipulation from Passive Human Videos
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 10:33 PM

  To overcome these challenges, we present mobile code-as-policy (MoCaP), an agentic framework that turns passive human video recordings into guidance for an end-to-end mobile manipulation policy.

  Passive human videos of everyday activity offer robots access to a massive amount of human experience without requiring a demonstration for each task.

Extractive summary: sentences quoted from the sources.