Thinker-WM: A Unified World Model with Shared Multimodal Representations for Embodied Intelligence

Key points

• World models have emerged as a promising paradigm for embodied AI, yet existing systems often remain fragmented in the modalities they model and the functions they support.1
• The model performs multimodal unification at the representation level: appearance, geometry, and action are transformed into a common dynamic representation and jointly evolved by shared parameters, while their distinct output structures are preserved.1
• To examine whether the shared representation captures action-related information, we introduce a lightweight motion probe that shows the frozen visual representation contains recoverable motion information.1
• Thinker-WM is trained on a heterogeneous corpus comprising approximately 20K hours of egocentric video, 10K hours of real-robot data, and 1K hours of simulation data, including approximately 4K hours of real-robot data with aligned depth supervision.1

Sources (1)

• [1]Thinker-WM: A Unified World Model with Shared Multimodal Representations for Embodied Intelligence
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 02:18 PM

  In this work, we present Thinker-WM, a unified multimodal world model that jointly models and generates RGB, metric depth, and robot action through a shared intermediate Unified Dynamics Transformer with modality-specific encoders and decoders.

  World models have emerged as a promising paradigm for embodied AI, yet existing systems often remain fragmented in the modalities they model and the functions they support.

Extractive summary: sentences quoted from the sources.