Prior-Fitted and Frozen Scorers for Multimodal Contrastive Learning

Key points

• Several such functions have been recently proposed to extend contrastive learning beyond two modalities.1
• Motivated by this broader view, we instantiate the scorer with a Transformer and find that even a randomly initialized model, when kept frozen, can outperform strong handcrafted multimodal contrastive baselines, whereas optimizing the scorer jointly with the encoders is less effective.1
• Building on this observation, we introduce the Prior-Fitted Scorer (PFS): a Transformer scorer pre-trained on synthetic multimodal data generated from structural causal model (SCM) priors before transferring it to real-world data.1
• More broadly, our results suggest that multimodal contrastive learning need not rely on manual similarity functions: Transformers can provide strong inductive biases, while causal pretraining can further shape these biases.1

Sources (1)

• [1]Prior-Fitted and Frozen Scorers for Multimodal Contrastive Learning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 10, 09:49 AM

  Multimodal contrastive learning typically relies on manually specified scoring functions, such as cosine similarity, to quantify the similarity between representations from different modalities.

  Several such functions have been recently proposed to extend contrastive learning beyond two modalities.

Extractive summary: sentences quoted from the sources.