A failure of Imagination: VLMs fail to use visual representations for text, but self-distillation can help

Key points

• We ask whether this alignment changes how language models represent text---especially on tasks where visualization would help, like geometry questions posed in words.1
• We distinguish weak multimodality (the model can answer questions about an input image) from strong multimodality (the model's text representations themselves become vision-aligned, improving performance on text-only tasks with latent visual structure).1
• We compare instruction-tuned Qwen-2.5 and Mistral models against their vision-aligned counterparts on the text variant of the task.1
• Probing experiments corroborate this: vision-aligned decoder embeddings carry no more visual structure than their text-only counterparts.1

Sources (1)

• [1]A failure of Imagination: VLMs fail to use visual representations for text, but self-distillation can help
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 18, 06:53 PM

  Vision-language (VL) models are typically built by fine-tuning a language model to attend to visual features in service of a Visual Question-Answering objective.

  We ask whether this alignment changes how language models represent text---especially on tasks where visualization would help, like geometry questions posed in words.

Extractive summary: sentences quoted from the sources.