Vision-Language Models Integrate Conflicting Evidence Like Biased Bayesian Observers

Key points

• Prior work has characterized the resulting behavior in terms of modality biases--e.g., a "blind faith in text"--but why preferences vary across models, domains, and input conditions remains unclear.1
• We develop a predictive account from a Bayesian perspective, positing that VLMs combine conditionally independent evidence from each modality with prior beliefs, subject to modality-specific biases.1
• The resulting two-parameter cue combination model accurately predicts VLM preferences under unseen combinations of input degradation and transfers to new forms of unreliability without refitting.1
• Separating evidence strength from integration bias explains how VLMs can follow text even while weighting visual evidence more heavily, reconciling reports of text and vision dominance under a single combination rule.1

Sources (1)

• [1]Vision-Language Models Integrate Conflicting Evidence Like Biased Bayesian Observers
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 01:55 AM

  Vision-language models must reconcile conflicting text and images.

  Prior work has characterized the resulting behavior in terms of modality biases--e.g., a "blind faith in text"--but why preferences vary across models, domains, and input conditions remains unclear.

Extractive summary: sentences quoted from the sources.