CheXFuse: Guided Differential Joint Attention Fusion for Robust Multimodal Chest X-Ray Classification in Data-Scarce Medical Settings

Key points

• A central challenge in multimodal learning is how to fuse modalities that differ substantially in predictive strength.1
• When one modality provides a much stronger representation than another, indiscriminate cross-modal interaction can introduce weak, redundant, or unstable signals, limiting or even degrading performance relative to the strongest unimodal representation.1
• CheXFuse combines image and text token representations within a joint multimodal sequence, where learned token guidance emphasizes informative representations and differential attention reduces less useful attention responses.1
• We evaluate CheXFuse on the Indiana University (IU) Chest X-ray dataset, using binary (normal vs. abnormal) classification as the downstream task.1

Sources (1)

• [1]CheXFuse: Guided Differential Joint Attention Fusion for Robust Multimodal Chest X-Ray Classification in Data-Scarce Medical Settings
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 12:45 AM

  To address this problem, we propose CheXFuse, a multimodal fusion framework based on Guided Differential Joint Attention (GuDJA).

  A central challenge in multimodal learning is how to fuse modalities that differ substantially in predictive strength.

Extractive summary: sentences quoted from the sources.