Towards a Data-Centric Understanding of Spatial Reasoning in Multimodal Large Language Models

Key points

• However, MLLMs suffer from limitations in spatial reasoning and perception.1
• In this survey, we consider approaches to fine-tuning pretrained MLLMs for spatial reasoning tasks, ranging from supervised to reinforcement learning, with an emphasis on the datasets used.1
• We investigate themes of dataset construction, data modality and annotation, and evaluation of the resulting models.1
• We find four main methods for constructing fixed spatial annotations: rules and templates, LLM prompting, procedural generation, and human-in-the-loop.1

Sources (1)

• [1]Towards a Data-Centric Understanding of Spatial Reasoning in Multimodal Large Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 07:42 PM

  Multimodal Large Language Models (MLLMs) have been widely applied in embodied agents and robotics for vision tasks.

  However, MLLMs suffer from limitations in spatial reasoning and perception.

Extractive summary: sentences quoted from the sources.