Stuck on Shuffle: On Attribute-Object Binding in Vision-Language Models

Key points

• Retrieval is a fundamental task in multimodal systems: given an image or text query, the goal is to find the most relevant images or texts in a collection, often using a shared embedding space.1
• One key compositional challenge is attribute--object binding: encoding which attributes belong to which objects.1
• Evaluating models on SABER reveals substantial binding failures in both CLIP-style dual encoders and recent MLLM-based embedders.1
• Despite the absence of image supervision, the gains transfer to cross-modal and image--image retrieval while preserving general multimodal embedding performance.1

Sources (2)

• [1]Stuck on Shuffle: On Attribute-Object Binding in Vision-Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 07:55 PM

  We therefore introduce SABER, a controlled benchmark for evaluating attribute--object binding across all four image and text modality pairings, spanning diverse attribute types and scenes with up to ten objects.

  Retrieval is a fundamental task in multimodal systems: given an image or text query, the goal is to find the most relevant images or texts in a collection, often using a shared embedding space.

• [2]Stuck on Shuffle: On Attribute-Object Binding in Vision-Language Models
  Hugging Face Daily Papers · Oct 15, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.