Slot-JEPA: Object-Structured Prediction Interfaces for Latent World Models

Key points

• However, dense patch representations expose a redundant interface to downstream predictors, which must implicitly infer spatial organization and temporal correspondence, presenting three challenges.1
• We propose the Slot-based Joint-Embedding Predictive Architecture (Slot-JEPA), which decomposes dense representations into stable semantic slots and time-varying spatial masks.1
• Building on it, our Transition Memory Predictor (TMP) stores visual transitions as compact masks and retrieves useful context using statistics of corresponding slots.1
• Slot-JEPA also improves goal-conditioned planning across six planar environments and four robotic arm planning tasks on the REALM-JEPA benchmark.1

Sources (2)

• [1]Slot-JEPA: Object-Structured Prediction Interfaces for Latent World Models
  Hugging Face Daily Papers · Oct 20, 12:00 AM

  As a prototypical latent world model, Joint-Embedding Predictive Architectures (JEPAs) provide strong semantic representations.

  However, dense patch representations expose a redundant interface to downstream predictors, which must implicitly infer spatial organization and temporal correspondence, presenting three challenges.

• [2]Slot-JEPA: Object-Structured Prediction Interfaces for Latent World Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 11:53 AM · same content

Extractive summary: sentences quoted from the sources.