Understanding Unified VLMs: From Attention Probing to Guidance Tuning for Multimodal Generation

Key points

• Unified vision-language models (VLMs) perform both text-to-image (T2I) and image-to-text (I2T) generation with one shared Transformer backbone, yet how the backbone processes the condition in either direction remains poorly understood.1
• We study this question from the perspective of conditional guidance: we probe every attention head to show the activation difference between two branches of classifier-free guidance (CFG) with different conditional guidance.1
• Experiments show consistent benefits in both multimodal generation directions of unified VLMs without additional training, and in particular substantial improvements over strong baselines in T2I generation.1
• These results further support the key finding of our probing analysis: the observed pattern of functional specialization is an intrinsic property of unified VLMs.1

Sources (1)

• [1]Understanding Unified VLMs: From Attention Probing to Guidance Tuning for Multimodal Generation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 10, 11:40 PM

  This characteristic pattern holds in both T2I and I2T generation and is largely stable across data samples, demonstrating the internal mechanism of unified VLMs. Motivated by this, we propose a new inference-time intervention approach for unified multimodal generation, named HeadTuned-CFG, which uses the probed condition-change awareness to select and tune the conditional guidance at the attention

  Unified vision-language models (VLMs) perform both text-to-image (T2I) and image-to-text (I2T) generation with one shared Transformer backbone, yet how the backbone processes the condition in either direction remains poorly understood.

Extractive summary: sentences quoted from the sources.