Expandable Modality Experts for Medical Vision-Language Learning

Key points

• Learning across medical imaging modalities requires making use of images without paired text and accommodating new domains while retaining existing representations.1
• A modality-aware visual tokenizer supplies targets for masked pretraining on 19.4 million unpaired image rows across seven modalities; contrastive alignment then uses 3.23 million image-text pairs.1
• Additional fundus and optical coherence tomography experts learn from paired data while the original image and text embedding paths remain frozen.1
• The proposed technique provides a modular approach to learning and expanding medical vision-language representations.1

Sources (1)

• [1]Expandable Modality Experts for Medical Vision-Language Learning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 03:28 PM

  We propose a medical vision-language technique built around an expandable bank of modality experts.

  Learning across medical imaging modalities requires making use of images without paired text and accommodating new domains while retaining existing representations.

Extractive summary: sentences quoted from the sources.