What Really Works for Universal Multimodal Embeddings? A Controlled Study of Contrastive Training Strategies

Key points

• Although various contrastive training strategies have been proposed across negative selection, batch construction, and gradient modulation, their actual effectiveness under controlled, reproducible conditions remains poorly understood.1
• This motivates a fundamental question: Do sophisticated contrastive strategies consistently bring meaningful downstream gains in multimodal embedding learning?1
• To uncover the underlying failure modes, we introduce an MLLM-as-a-Judge diagnostic framework that dissects top-1 retrieval behaviors into labeled positives, false negatives (unlabeled valid matches), and hard/easy negative errors.1
• Overall, our findings demystify popular contrastive strategies and provide actionable recipes for training universal multimodal embedding models.1

Sources (2)

• [1]What Really Works for Universal Multimodal Embeddings? A Controlled Study of Contrastive Training Strategies
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 10, 04:19 PM

  Contrastive learning has become the standard paradigm for multimodal representation learning, aligning semantically matched pairs while separating mismatched ones in a shared embedding space.

  Although various contrastive training strategies have been proposed across negative selection, batch construction, and gradient modulation, their actual effectiveness under controlled, reproducible conditions remains poorly understood.

• [2]What Really Works for Universal Multimodal Embeddings? A Controlled Study of Contrastive Training Strategies
  Hugging Face Daily Papers · Oct 10, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.