ViCoR: Estimating Visual Necessity via Counterfactual Residuals for Multimodal Medical Data Selection

Key points

• Medical vision-language models often exhibit hidden stratification, where high overall performance masks failures on subtle visual findings and overlapping clinical conditions.1
• We trace this failure mode to modality imbalance during fine-tuning, where strong linguistic priors can suppress visually grounded learning and induce plausible but weakly grounded rationales.1
• ViCoR estimates visual necessity through image-dependent optimization residuals, calibrates the resulting scores with medical supervision compatibility, and constructs a diversity-aware fine-tuning subset.1
• Gradient-level analysis and perturbation audits further show that ViCoR enriches image-dependent training signals and strengthens image-level and evidence-region dependence.1

Sources (1)

• [1]ViCoR: Estimating Visual Necessity via Counterfactual Residuals for Multimodal Medical Data Selection
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 18, 11:46 AM

  To address this problem, we propose ViCoR, a counterfactual-residual-based data selection method for Med-VLM fine-tuning.

  Medical vision-language models often exhibit hidden stratification, where high overall performance masks failures on subtle visual findings and overlapping clinical conditions.

Extractive summary: sentences quoted from the sources.