Domain-specific language alignment of multimodal embeddings towards explainable predictions

Key points

• Multimodal data are commonly fused into latent representations that improve downstream task performance but are not interpretable, limiting their application in domains that require explainable model decisions.1
• Vision-language alignment has made image embeddings interpretable through language, but requires vast image-caption training data.1
• Here, we propose a framework to language-align any embedding space that can be linked to interpretable auxiliary data.1
• AlphaEarth embeddings beat previous state-of-the-art models on 6/7 metrics, while our framework language-aligns domain-specific concepts of the embedding space, outperforming previous Earth observation vision-language models on average concept retrieval for both datasets.1

Sources (1)

• [1]Domain-specific language alignment of multimodal embeddings towards explainable predictions
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 08:20 AM

  We apply this framework to multimodal Earth embeddings to facilitate explainable predictions for two species distribution modelling tasks: S2BMS and SatBird.

  Multimodal data are commonly fused into latent representations that improve downstream task performance but are not interpretable, limiting their application in domains that require explainable model decisions.

Extractive summary: sentences quoted from the sources.