Compositionality with Primitives: A Unified View from Euclidean to Hyperbolic Vision-Language Models

Key points

• Prior work showed that large-scale vision-language models encode decomposable semantic structures that can be recomposed to enable generalization.1
• The Geodesically Decomposable Embeddings (GDE) framework formalizes this by identifying directions in the tangent space at the Frechet mean of the embedding distribution, which encode primitive visual concepts.1
• By means of new compositional scoring functions and extensive evaluations, we show that directly using the decomposed primitives suffices for recognizing unseen compositions.1
• Across models (CLIP, DINOv2, MERU, HyCoCLIP), we show that compositional structure consistently emerges as primitive representations, enabling inference on CZSL benchmarks (MIT-States, VAW-CZSL, UT-Zappos50K, Clothing16K) and group robustness on Waterbirds and CelebA.1

Sources (1)

• [1]Compositionality with Primitives: A Unified View from Euclidean to Hyperbolic Vision-Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 02:34 PM

  Compositional zero-shot learning (CZSL) aims to recognize novel combinations of previously observed objects and attributes.

  Prior work showed that large-scale vision-language models encode decomposable semantic structures that can be recomposed to enable generalization.

Extractive summary: sentences quoted from the sources.