MSCR: Jointly Balancing Utilization Across Modalities and Discovering Synergistic Information

Key points

• Balance-oriented methods such as the Multimodal Competition Regularizer (MCR) perform well by alleviating this imbalance during training.1
• These dynamic balancing methods, however, mainly adjust modality contributions and do not consider how imbalance affects the formation of synergistic information during fusion.1
• We propose the Multimodal Synergistic Competition Regularizer (MSCR), a fusion method that jointly promotes balanced modality utilization and the discovery of synergistic information.1
• An anti-dominance objective suppresses excessive unimodal influence and leaves room for synergistic information, and a synergy objective encourages the fused predictor to outperform a non-synergistic unimodal reference.1

Sources (1)

• [1]MSCR: Jointly Balancing Utilization Across Modalities and Discovering Synergistic Information
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 17, 02:54 AM

  In multimodal learning, modality imbalance often lets one modality dominate optimization and keeps the model from fully exploiting cross-modal interactions.

  Balance-oriented methods such as the Multimodal Competition Regularizer (MCR) perform well by alleviating this imbalance during training.

Extractive summary: sentences quoted from the sources.