From Pixels to Executable Physics: Behavior-Grounded World Models for Embodied Action Consequence Prediction

Key points

• In outdoor environments, varying terrain and contact conditions can cause identical action commands to produce different motion outcomes for quadruped robots.1
• Existing video world models can generate visually plausible futures, yet often struggle to capture robot dynamics and terrain interactions, leading to overly optimistic predictions of action outcomes.1
• Starting from an initial RGB image, our method reconstructs scene geometry and integrates dense, dimensionless priors over terrain contact properties, calibrated using real-world action-response data.1
• These physically grounded predictions support more informed action selection: across 90 challenging scenarios, an 8B vision-language model augmented with DR-WM outperforms an unassisted 32B vision-language model on five of six evaluation metrics.1

Sources (1)

• [1]From Pixels to Executable Physics: Behavior-Grounded World Models for Embodied Action Consequence Prediction
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 02:56 AM

  We present DR-WM, a physically consistent world model that incorporates robot-terrain contact dynamics to predict the consequences of quadruped actions.

  In outdoor environments, varying terrain and contact conditions can cause identical action commands to produce different motion outcomes for quadruped robots.

Extractive summary: sentences quoted from the sources.