Persistent Representational Misalignment Induces Vision-Language Model Failures

Key points

• Vision-language models (VLMs) fail at some tasks that are simple for humans---but why?1
• We hypothesize that many such failures stem from VLMs' difficulties translating between vision and language representations, a misalignment the Platonic Representation Hypothesis predicts should disappear with scale.1
• In the second, we demonstrate how this misalignment both leads to a predictably induced generalization failure and correlates with existing failure modes, where more representationally aligned VLM backbones accordingly exhibit fewer failures.1
• Overall, these results imply representations will not become aligned simply given sufficient scale, providing evidence against a strong version of the Platonic Representation Hypothesis.1

Sources (1)

• [1]Persistent Representational Misalignment Induces Vision-Language Model Failures
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 18, 01:11 PM

  This paper is composed of two parts: in the first, we show that existing vision and language representations are linearly unalignable.

  Vision-language models (VLMs) fail at some tasks that are simple for humans---but why?

Extractive summary: sentences quoted from the sources.