TartanMatch: Towards Universal Dense Correspondence Across Modalities

Key points

• Robotic platforms combine multi-modal sensors to operate across diverse conditions, but most dense correspondence methods are limited to RGB-RGB or a small set of modality pairs.1
• TartanMatch adapts an RGB-pretrained correspondence model to all modality pairs within one shared network using lightweight modality adaptation and a mixture of real and synthetic data.1
• On zero-shot real-world benchmarks, TartanMatch reduces Endpoint Error (EPE) over the strongest baselines by 61.9% for cross-modal matching and 49.6% for same-modal matching.1
• We further show that joint training across all modality pairs improves average matching accuracy over pair-specific training, especially on challenging cross-modal pairs.1

Sources (1)

• [1]TartanMatch: Towards Universal Dense Correspondence Across Modalities
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 10, 03:19 AM

  We introduce TartanMatch, a unified model for dense correspondence across all 25 ordered pairs among RGB, event, thermal, depth, and projected LiDAR.

  Robotic platforms combine multi-modal sensors to operate across diverse conditions, but most dense correspondence methods are limited to RGB-RGB or a small set of modality pairs.

Extractive summary: sentences quoted from the sources.