Learning to Think and Explore in 3D World

Key points

• Instead of providing multi-view images, videos, or 3D representations to the model, we preserve the native visual encoding of VLMs, i.e., a single image, one at a time, just like humans, and treat the 3D representation only as an environment with which the model can interact through actions (e.g., move left) and observations (rendered images).1
• Following an initial trajectory-supervised finetuning, we apply reinforcement learning (RL) with rewards driven primarily by final-answer correctness.1
• Further analysis reveals adaptive, question-conditioned exploration.1
• ACT3D searches for occluded targets, avoids unnecessary exploration, and acquires complementary viewpoints for spatial comparisons.1

Sources (2)

• [1]Learning to Think and Explore in 3D World
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 12:38 PM

  We study how vision-language models (VLMs) can learn to think and actively explore in 3D space.

  Instead of providing multi-view images, videos, or 3D representations to the model, we preserve the native visual encoding of VLMs, i.e., a single image, one at a time, just like humans, and treat the 3D representation only as an environment with which the model can interact through actions (e.g., move left) and observations (rendered images).

• [2]Learning to Think and Explore in 3D World
  Hugging Face Daily Papers · Oct 15, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.