Foundation-Model-Driven 3D Scene Graphs for Mobile Manipulation

Key points

• Recent work (e.g., FOUND-IT) has leveraged these tools to construct real-time, task-driven, and open-set 3D scene graphs from a single uncalibrated camera.1
• However, while classical robot mapping techniques have undergone extensive system demonstrations, limited work has been done to develop both the fundamental and the systems wide advancements needed to integrate these new types of robot spatial memory into a full, useable autonomy pipeline.1
• Towards this end, we integrate FOUND-IT into a mobile manipulator platform and demonstrate it is well-suited to support multiple objectives such as object change detection, support for multiple mapping sessions, determination of navigable paths, and directly support object grasping for the task of scene re-arrangement.1
• Furthermore, we provide nuanced insights into the effectiveness and limitations of a monocular, foundation-model-driven 3D scene graph construction system using observations from rigorous on-robot experiments.1

Sources (1)

• [1]Foundation-Model-Driven 3D Scene Graphs for Mobile Manipulation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 03:32 PM

  Geometric and Semantic Foundation models are becoming increasingly popular tools for generating spatial maps for robots.

  Recent work (e.g., FOUND-IT) has leveraged these tools to construct real-time, task-driven, and open-set 3D scene graphs from a single uncalibrated camera.

Extractive summary: sentences quoted from the sources.