Design Choices for Multimodal Transformer Learning with Missing and Longitudinal Healthcare Data

Key points

• Multimodal learning offers the potential to improve predictions by integrating complementary information from heterogeneous data sources.1
• In this study, we systematically investigate these design choices using three different cancer datasets.1
• Experiments are conducted with unimodal and multimodal configurations, as well as different levels of missingness for binary survival classification and multiple time encoding strategies for discrete-time survival prediction.1
• Our results show that while multimodal outperforms unimodal results, missing data heavily impacts predictions and thus effective design choices such as tokenization granularity, and intra-missing handling is essential for reliable prediction.1

Sources (1)

• [1]Design Choices for Multimodal Transformer Learning with Missing and Longitudinal Healthcare Data
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 10, 01:05 AM

  Although recent approaches increasingly rely on Transformer-based architectures for multimodal fusion, incomplete multimodal data introduces challenges such as missing modalities and intra-modal missingness, which motivate key architectural design choices, such as tokenization granularity.

  Multimodal learning offers the potential to improve predictions by integrating complementary information from heterogeneous data sources.

Extractive summary: sentences quoted from the sources.