Exploring Temporal Context-Aware Physical Grounding in Latent World Models

Key points

• Learning latent dynamics is fundamental to building action-conditioned world models for planning.1
• Among existing families, Joint Embedding Predictive Architectures (JEPAs) offer a natural framework by modeling transitions directly in latent space rather than at the pixel level.1
• Specifically, during learning, we incorporate a shared causal temporal module that aggregates temporal context within observation and prediction sequences under physical state supervision, grounding both in task-relevant physical states.1
• When planning, we further introduce action inference through stepwise conditioning of a generative action distribution on the current and goal latent representations together with their evolving discrepancy, yielding anchored proposals that guide model-based online refinement.1

Sources (1)

• [1]Exploring Temporal Context-Aware Physical Grounding in Latent World Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 04:30 PM

  To this end, we propose $\mathcal{G}$-JEPA, which incorporates temporal context into physical grounding for latent world models and couples the learned models with prior-guided action planning.

  Learning latent dynamics is fundamental to building action-conditioned world models for planning.

Extractive summary: sentences quoted from the sources.