VGL-stGen: Learning a shared multimodal model for lung histology and spatial transcriptomics
VGL-stGen is a multimodal large language model adapted from a pretrained medical vision-language model using paired lung H&E and spatial transcriptomics with single-cell references.
ProofPaper ↗
Key points
- Training integrates image-expression and expression-cell-type pairs, and gene-conditioned pseudo-labels transfer transcriptomic cell-type supervision to histology images.
- One deployed checkpoint supports thirteen evaluated tasks spanning tissue and cell-type classification, gene prediction and conditional expression completion from combinations of histology, cell identity and observed gene expression.
- Task-specific decoding produces categorical labels, gene lists and structured expression levels.
- Image-permutation experiments indicate that correctly paired histology adds information beyond observed expression.
Sources (1)
- [1]VGL-stGen: Learning a shared multimodal model for lung histology and spatial transcriptomicsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 02:22 PM
VGL-stGen is a multimodal large language model adapted from a pretrained medical vision-language model using paired lung H&E and spatial transcriptomics with single-cell references.
Training integrates image-expression and expression-cell-type pairs, and gene-conditioned pseudo-labels transfer transcriptomic cell-type supervision to histology images.
Extractive summary: sentences quoted from the sources.