Thinking with Images without Coordinates

Key points

• We identify two fundamental limitations in this approach, i.e., a semantic gap where sparse regressions miss high-dimensional latent structures, and a resolution gap due to the granularity mismatch between patch-level tokens and pixel-level coordinates.1
• To address these, we propose TiCO (Thinking with images without COordinates).1
• It reformulates visual reasoning as a semantic evidence discovery process within the latent space, and evidence is injected as token-level regions without pixel coordinates.1
• Extensive experiments across multiple high-resolution and general visual reasoning benchmarks demonstrate that our TiCO consistently outperforms state-of-the-art coordinate-based methods with significantly improved inference efficiency.1

Sources (1)

• [1]Thinking with Images without Coordinates
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 07:38 PM

  Recent advances in multi-modal large language models have shifted toward the "thinking with images" paradigm to improve reasoning faithfulness, yet existing methods rely heavily on explicit coordinate- based grounding.

  We identify two fundamental limitations in this approach, i.e., a semantic gap where sparse regressions miss high-dimensional latent structures, and a resolution gap due to the granularity mismatch between patch-level tokens and pixel-level coordinates.

Extractive summary: sentences quoted from the sources.