LeWAM: End-to-End World and Action Modeling with JEPAs
We introduce LeWAM, an end-to-end JEPA for control with: (1) a joint training objective over next latent state prediction and action flow matching, and (2) an isotropic Gaussian constraint on the latent state, enabling a single model to support direct policy execution and efficient planning by evaluating sampled actions with its learned latent dynamics.
ProofPaper ↗
Key points
- Joint-embedding predictive architectures (JEPAs) learn compact latent world models without pixel reconstruction.
- Existing JEPA pipelines typically train visual representations and action-conditioned dynamics before adding a control policy or solver.
- Compared with prior JEPAs in visual planning, LeWAM improves success rate from 28.6% to 89.7% on contact-rich tasks and from 10.9% to 46.2% on long-horizon tasks, and plans 32.3 times faster.
- Interpretability analyses show that LeWAM learns control-relevant representations and action distributions effectively guiding test-time planning and control.
Sources (2)
- [1]LeWAM: End-to-End World and Action Modeling with JEPAsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 11:58 PM
We introduce LeWAM, an end-to-end JEPA for control with: (1) a joint training objective over next latent state prediction and action flow matching, and (2) an isotropic Gaussian constraint on the latent state, enabling a single model to support direct policy execution and efficient planning by evaluating sampled actions with its learned latent dynamics.
Joint-embedding predictive architectures (JEPAs) learn compact latent world models without pixel reconstruction.
- [2]LeWAM: End-to-End World and Action Modeling with JEPAsHugging Face Daily Papers · Oct 8, 12:00 AM · same content
Extractive summary: sentences quoted from the sources.
Before this
- Oct 7, 2026Sparse Planning in Visual World Models via Cost Gradients
- Oct 7, 2026UltraWorld: Learning Interactive Ultrasound World Models from Untracked Clinical Videos with Acoustic Sampling Map
- Oct 7, 2026Controllable Crowd Generation through World-Model Planning
- Oct 7, 2026RoboJEPA: Scaling Laws for Multi-Embodiment Robotic Latent World Models
- Oct 7, 2026DSReg: Provably Recovering Individual World Latents without Reconstruction
- Oct 6, 2026Independent Multi-Agent Reinforcement Learning with Counterfactual Semantic-Social World Models