Can World and Action Modeling Benefit Each Other under Unified Causal Training?

Key points

• World and action modeling are closely related, yet their reciprocal benefits under unified causal training remain unclear.1
• Using this framework, we investigate whether learning forward dynamics conditioned on specified actions improves action learning, and whether learning to generate actions in turn improves action-conditioned world modeling.1
• We use LIBERO, LIBERO-Plus, and RoboCasa365 as embodied testbeds to evaluate both policy learning and world modeling, complemented by real-world policy experiments.1
• In turn, action learning improves multi-view world-prediction fidelity under the same architecture.1

Sources (2)

• [1]Can World and Action Modeling Benefit Each Other under Unified Causal Training?
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 01:53 AM

  We introduce OneWM, a simple framework for unified causal training that treats actions as both conditions for world prediction and targets for policy learning within a shared video-action architecture.

  World and action modeling are closely related, yet their reciprocal benefits under unified causal training remain unclear.

• [2]Can World and Action Modeling Benefit Each Other under Unified Causal Training?
  Hugging Face Daily Papers · Oct 15, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.