Can Visual Generation Expand What Unified Multimodal Models Know?

Key points

• Unified multimodal models (UMMs) receive image-generation supervision that exposes them to visual information not fully documented in text.1
• Can learning to generate images expand what pretrained models know?1
• Motivated by these observations, we successively introduce Textual Alignment and Interleaved Image--Text Alignment, which connect generation and answering through shared knowledge and interleaved responses, respectively.1
• These findings support visual generation as a source of knowledge beyond textual supervision, and motivate joint investigation of training data design and model architecture to make this knowledge more broadly accessible.1

Sources (2)

• [1]Can Visual Generation Expand What Unified Multimodal Models Know?
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 08:03 AM

  We introduce Knowledge Transfer Probing (KTP), a controlled protocol that supplies new knowledge through image-generation training and tests its accessibility through text-only question answering.

  Unified multimodal models (UMMs) receive image-generation supervision that exposes them to visual information not fully documented in text.

• [2]Can Visual Generation Expand What Unified Multimodal Models Know?
  Hugging Face Daily Papers · Oct 15, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.