Seeing is Believing: Benchmarking and Mitigating the Interference of Language Priors on Visual Perception in MLLMs

Key points

• Existing benchmarks assume shared priors across models and rely solely on accuracy.1
• To address these limitations, we introduce ViSiP-Bench, a benchmark grounded in rare-but-real scenarios across six prior categories, with a model-aware dynamic prior induction protocol and three metrics that more precisely and continuously measure prior resistance than accuracy alone.1
• Evaluating 24 models reveals that prior adherence is pervasive, does not improve with scale, and larger models tend to either strongly override their priors or firmly adhere to them.1
• We therefore propose MAPS, which identifies each model's strongest priors and fine-tunes on factually grounded knowledge to soften excessive prior reliance without disrupting general capabilities.1

Sources (1)

• [1]Seeing is Believing: Benchmarking and Mitigating the Interference of Language Priors on Visual Perception in MLLMs
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 01:54 PM

  Language priors in MLLMs can override visual evidence, impairing visual perception.

  Existing benchmarks assume shared priors across models and rely solely on accuracy.

Extractive summary: sentences quoted from the sources.