LocaLeJEPA: Maximum Entropy Representations under Geometric Priors

Key points

• Joint-Embedding Predictive Architectures (JEPA) offer an appealing approach to self-supervised learning, but are typically prone to representation collapse.1
• LeJEPA resolves this by introducing a theoretically grounded distribution-matching objective that drives the embedding towards an isotropic Gaussian.1
• The accompanying LocaLeReg, a novel locally Euclidean distribution-matching regularizer, enables a stable and scalable implementation of the LocaLeJEPA framework.1
• We view the LocaLeJEPA formalism as a principled approach to the JEPA paradigm that better respects intrinsic data geometry while also accommodating generalizations to exotic embedding geometries.1

Sources (1)

• [1]LocaLeJEPA: Maximum Entropy Representations under Geometric Priors
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 04:15 PM

  We thus introduce LocaLeJEPA, a framework motivated by a manifold-theoretic reformulation of LeJEPA's entropy-maximization property.

  Joint-Embedding Predictive Architectures (JEPA) offer an appealing approach to self-supervised learning, but are typically prone to representation collapse.

Extractive summary: sentences quoted from the sources.