Text-Anchored Modality Experts with Uncertainty-Guided Routing for Multimodal Sentiment Analysis

Key points

• Multimodal sentiment analysis (MSA) aims to understand user-generated videos by integrating multiple modalities.1
• Specifically, TAME adopts an asymmetric Mixture-of-Modality-Experts framework, in which textual features act as a fixed and always-activated shared expert, while audio-textual and video-textual cross-modal branches serve as dynamically routed auxiliary experts.1
• Furthermore, we introduce lightweight Prophet Heads to quantify the sample-level predictive uncertainty of each individual modality, and this uncertainty metric is further incorporated into the modality router to realize reliability-aware expert aggregation and fusion.1
• Experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS datasets demonstrate that TAME achieves state-of-the-art performance on the majority of metrics while reducing parameter count by up to 78.7% and FLOPs by up to 41.2% over the mixture-of-expert baseline.1

Sources (1)

• [1]Text-Anchored Modality Experts with Uncertainty-Guided Routing for Multimodal Sentiment Analysis
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 03:39 AM

  To address this, we propose a Text-Anchored Modality Experts with Uncertainty-Guided Routing, termed TAME, that reformulates MSA as an uncertainty-guided refinement paradigm, where the language prior is calibrated and enhanced by selectively trusted nonverbal evidence.

  Multimodal sentiment analysis (MSA) aims to understand user-generated videos by integrating multiple modalities.

Extractive summary: sentences quoted from the sources.