Perceive, Predict, and Plan: a Unified World Action Model for Egocentric Dexterous Manipulation

Key points

• Randomized modality visibility lets video and 3D actions serve as either conditioning context or prediction targets across the three modes.1
• Experiments show that unified training improves all three capabilities over single-mode training, including gains in planning quality under a vision-language model evaluation.1
• Through unified training, our model outperforms specialized baselines on 3D hand reconstruction and action-conditioned video prediction across four datasets, while producing goal-directed manipulation videos with stronger instruction adherence and action progression than video-generation baselines.1

Sources (1)

• [1]Perceive, Predict, and Plan: a Unified World Action Model for Egocentric Dexterous Manipulation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 05:59 PM

  Our model integrates a compact, structured 3D action representation into a pretrained video diffusion transformer through a mixture-of-transformers architecture with shared base weights, modality-specific adaptation, and bidirectional visual-action attention.

  Randomized modality visibility lets video and 3D actions serve as either conditioning context or prediction targets across the three modes.

Extractive summary: sentences quoted from the sources.