Beyond Sparse Captions: Aligning Slide-Level Text and Patch-Level Vision in Pathology

Key points

• We propose to instead leverage slide-level pathology reports which can be generated during routine diagnostic workflows in a scalable manner.1
• We introduce Locked Image Multiple Instance PreTraining (LIMIT), which addresses this gap by freezing a pretrained vision encoder and fine-tuning only the text encoder: report embeddings serve as cross-attention queries over bags of patch features, producing report-conditioned slide representations optimized via InfoNCE.1
• Using LIMIT, we establish CONCH-Z, which aligns the text encoder of CONCH v1.5 to 335k clinical reports.1
• Evaluated across 22 tasks spanning patch-level classification, tumor detection, and slide-level subtyping, CONCH-Z establishes state-of-the-art zero-shot performance over VL encoders trained on curated caption datasets, while simultaneously closing the gap with slide-level foundation models trained with substantially more parameters and compute.1

Sources (1)

• [1]Beyond Sparse Captions: Aligning Slide-Level Text and Patch-Level Vision in Pathology
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 06:03 PM

  Contrastive vision-language (VL) pretraining in computational pathology has generally relied on curated image-caption datasets sourced from social media, educational videos, and research publications.

  We propose to instead leverage slide-level pathology reports which can be generated during routine diagnostic workflows in a scalable manner.

Extractive summary: sentences quoted from the sources.