LaLa-World: Scaling Latent World Models with Natural Language Actions

Key points

• This paper shows that JEPA-style latent world models (LWMs) can learn across diverse domains and data sources by adopting the language-conditioned interface of video generation models.1
• We establish this approach as an effective and efficient basis for planning with imagination and downstream policy fine-tuning.1
• In a matched comparison with an RAEv2-based video generation model baseline, LaLa-World reaches comparable planning performance with $6\times$ less training cost and $13.9\times$ faster planning.1
• Beyond planning, the pretrained LaLa-World serves as an initialization for LaLa-Policy, which jointly predicts actions and future visual representations.1

Sources (1)

• [1]LaLa-World: Scaling Latent World Models with Natural Language Actions
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 08:32 AM

  To this end, we introduce LaLa-World, a Language-conditioned Latent World model that learns from simulation and robot trajectories together with broader image and video data.

  This paper shows that JEPA-style latent world models (LWMs) can learn across diverse domains and data sources by adopting the language-conditioned interface of video generation models.

Extractive summary: sentences quoted from the sources.