Visual Scaffolding: Toward Native Multimodal Reasoning

Key points

• We therefore ask: Can native visual representations become first-class citizens alongside language, providing a visual scaffold for reasoning?1
• We define native multimodal reasoning as reasoning in which native visual representations and language jointly carry reusable intermediate states.1
• As a concrete implementation, the visual scaffolding mechanism we proposed lets the MLLM autonomously initiate a retrieval action at any reasoning step.1
• To evaluate this mechanism under controlled visual demands, we introduce NMR-Bench, comprising Burden Counting and Path Tracing.1

Sources (1)

• [1]Visual Scaffolding: Toward Native Multimodal Reasoning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 18, 01:06 PM

  The success of chain-of-thought (CoT) in complex reasoning can be attributed to a textual scaffolding effect: explicit intermediate results can be examined, reused, and composed to support subsequent reasoning, much as scaffolding supports a building under construction.

  We therefore ask: Can native visual representations become first-class citizens alongside language, providing a visual scaffold for reasoning?

Extractive summary: sentences quoted from the sources.