Causal Analysis of Robot Foundation Models Enables Efficiency Gains

Key points

• While the internal computation of vision-language models, which compose the backbone of VLAs, has been studied extensively, less is known about how their perceptual and semantic representations are transformed into action in VLA models.1
• We study this question mechanistically in three frontier VLA models ($π{0.5}$, MolmoAct2, and MolmoBot) by analyzing the causal pathways through which the input image and the instruction shape actions.1
• Across the three models, activation patching shows that causally relevant information influences the action expert (AE) through a narrow band of mid-to-late cross-attention layers, which we term the action-conditioning interface.1
• We find that restricting MolmoBot's action expert conditioning to only 3 out of 36 layers reduces inference FLOPs by up to $33%$ while maintaining comparable task success with the default model across 3,000 evaluation episodes ($46.7%$ vs. $46.9%$), and the same principle carries over to real-robot deployment of $π{0.5}$.1

Sources (1)

• [1]Causal Analysis of Robot Foundation Models Enables Efficiency Gains
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 06:06 PM

  Vision-language-action (VLA) models are a promising approach to general purpose robot manipulation.

  While the internal computation of vision-language models, which compose the backbone of VLAs, has been studied extensively, less is known about how their perceptual and semantic representations are transformed into action in VLA models.

Extractive summary: sentences quoted from the sources.