Rethinking Projector Training in Multimodal LLMs

Key points

• Despite its widespread use, the role of the projector training stage remains poorly understood.1
• We find that (i) using only $10$ to $20%$ of the data for training the projector already recovers most downstream performance gains, and (ii) projectors trained on different datasets can be linearly interpolated while largely preserving performance.1
• Motivated by these findings, we propose PORTAL, a training-free projector initialization that requires neither paired image-caption data nor gradient-based optimization, but only per-modality summary statistics.1
• PORTAL computes an optimal-transport-based map between Gaussian approximations of the vision and language feature distributions on a shared principal subspace, so that the projector training stage can be skipped entirely.1

Sources (2)

• [1]Rethinking Projector Training in Multimodal LLMs
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 03:53 PM

  Multimodal LLMs are typically trained by first aligning a vision encoder with a language model via projector training on large image-caption datasets, followed by joint training of the full multimodal model.

  Despite its widespread use, the role of the projector training stage remains poorly understood.

• [2]Rethinking Projector Training in Multimodal LLMs
  Hugging Face Daily Papers · Oct 13, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.