Large Multimodal Models for Direct Robot Control: GPT-6 as a Beginning

Key points

• This paper systematically evaluate GPT-6 Astra for closed-loop robotic manipulation using a lightweight agent execution framework.1
• We first probe three fundamental embodied capabilities-depth perception, embodied in-context learning, and tactile understanding-and demonstrate that the model can exploit depth information, acquire simple manipulation strategies from contextual demonstrations, and use tactile feedback to correct its actions.1
• Despite competitive performance on several benchmarks-26.92% success on RoboDojo and 76.0% on RoboTwin 2.0-our analysis reveals a pronounced gap between semantic reasoning and physical execution, particularly in consequence-aware action planning, precision, and complex motion coordination.1
• To further examine these limitations, we introduce LIBERO-GeoMotion, a new benchmark of geometry-constrained tasks that require precise pose alignment, coordinated translation and rotation, and anticipation of physical action consequences.1

Sources (1)

• [1]Large Multimodal Models for Direct Robot Control: GPT-6 as a Beginning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 02:15 PM

  Recent advances in visuospatial understanding and agentic reasoning enable direct robotic manipulation with general-purpose multimodal models, but their embodied capabilities remain poorly understood.

  This paper systematically evaluate GPT-6 Astra for closed-loop robotic manipulation using a lightweight agent execution framework.

Extractive summary: sentences quoted from the sources.