Unifying Embodied Foundation Models via a Model-to-Policy Formulation

Key points

• Vision-language-action (VLA) models and world-action models (WAMs) have emerged as promising approaches to robot learning and control.1
• These models integrate multimodal perception and action generation, but differ in their architectures, output representations, and learning objectives.1
• Our formulation has two key components: (i) we characterize embodied foundation models through conditional output distributions and action-realization kernels, which together induce task-conditioned robot policies; and (ii) we connect pretraining and post-training to policy deployment, accommodating model-specific pretraining objectives and offline or online post-training strategies.1
• We instantiate the formulation for OpenVLA and DreamZero, covering autoregressive action-token prediction and joint video-action generation.1

Sources (1)

• [1]Unifying Embodied Foundation Models via a Model-to-Policy Formulation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 07:30 AM

  In this paper, we present a unified, architecture-agnostic probabilistic formulation for embodied foundation models based on stochastic kernels.

  Vision-language-action (VLA) models and world-action models (WAMs) have emerged as promising approaches to robot learning and control.

Extractive summary: sentences quoted from the sources.