EDMA: Entropy-Driven Multimodal Answering

Key points

• Multimodal question answering (MMQA) is based on integrating heterogeneous data sources, selectively leveraging relevant modalities, and ignoring distractors.1
• We argue that quantifying the effect that each modality has on the answering process improves process traceability and final performance.1
• EDMA introduces (i) modality-conditioned partitioning to estimate unimodal entropy reduction, (ii) entropy-driven multimodal fusion to capture complementary information across modalities, and (iii) entropy-based selection of the answering trajectory that minimises the most logical entropy.1
• Moreover, we show that higher entropy correlates with more false positives and, through controlled trajectory selection, demonstrate that logical entropy serves as an effective, monotonic selector for correct reasoning paths, validating entropy level as a reliable proxy for performance.1

Sources (1)

• [1]EDMA: Entropy-Driven Multimodal Answering
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 06:18 PM

  Thus, we propose Entropy-Driven Multimodal Answering (EDMA), which uses logical entropy to formalise multimodal QA as an iterative process of entropy reduction and to quantify the contribution of each modality in distinguishing correct from incorrect candidate answers.

  Multimodal question answering (MMQA) is based on integrating heterogeneous data sources, selectively leveraging relevant modalities, and ignoring distractors.

Extractive summary: sentences quoted from the sources.