MIST: Multi-Scale Spatial Context for Gene Expression Prediction from Histology

Key points

• A standard Transformer applies self-attention over spots but ignores where each spot sits in the tissue, whereas a spot's molecular state depends on tissue context at multiple spatial scales.1
• Building on this insight, we introduce MIST, a Multi-context Spatial Transformer that augments the self-attention backbone with two context pathways applied in every layer: a coordinate-invariant local pathway that attends over spatial k-nearest-neighbor spots, and a slide pathway that broadcasts a whole-slide summary to every spot; the model predicts all spots in a single forward pass.1
• In intra-cohort evaluation across six human organ cohorts from HEST-1k, MIST improves mean Pearson correlation over the strongest evaluated baseline by 9.7%, 11.6%, and 11.4% for 10-, 50-, and 100-gene panels, respectively, with the lowest average MSE and MAE.1
• Together, these results show that adding local and slide context to a vanilla Transformer backbone is an effective approach to spatial gene expression prediction from histology.1

Sources (1)

• [1]MIST: Multi-Scale Spatial Context for Gene Expression Prediction from Histology
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 05:29 PM

  Predicting spatial gene expression from standard H&E images could extend molecular analysis to tissues without spatial transcriptomic measurements.

  A standard Transformer applies self-attention over spots but ignores where each spot sits in the tissue, whereas a spot's molecular state depends on tissue context at multiple spatial scales.

Extractive summary: sentences quoted from the sources.