MERP: A Latent-Bottleneck Framework for Robust Multi-Modal Multi-Label Emotion Recognition

Key points

• Multimodal multi-label emotion recognition is important for affect-aware systems, but models trained with complete text, visual, and acoustic streams often degrade sharply when some streams are unavailable at deployment.1
• We address this train-test gap by learning a compact representation that preserves emotion-specific evidence from any available subset of modalities.1
• MERP further uses positive and negative class prototypes, a noise-gated feature enhancer with frequency-shaped perturbations during training, and confidence-aware class-wise fusion to strengthen separability, reduce sensitivity to missing inputs, and combine complementary predictors.1
• These results show that fixed latent reasoning provides a practical, testable route to robust multimodal emotion recognition without requiring missing-modality examples during training.1

Sources (1)

• [1]MERP: A Latent-Bottleneck Framework for Robust Multi-Modal Multi-Label Emotion Recognition
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 08:54 AM

  We propose the Multi-modal Emotion Recognition Perceiver (MERP), which maps the joint sequence into a fixed-size latent bank, thereby keeping repeated reasoning independent of raw input length; it extracts label-specific evidence before and after latent reasoning and aligns the two levels through reconstruction.

  Multimodal multi-label emotion recognition is important for affect-aware systems, but models trained with complete text, visual, and acoustic streams often degrade sharply when some streams are unavailable at deployment.

Extractive summary: sentences quoted from the sources.