Multi-Dimensional Featurizers Yield Fine-Grained Steering and Data Shaping in Robot Foundation Models

Key points

• Recent work in vision and language models reveals multidimensional features, motivating block-sparse featurizers (BSFs), which represent each feature in a low-dimensional subspace.1
• We show that BSFs recover several multi-dimensional task-relevant features underlying the model's computation (e.g., perception, language, proprioceptive state, episode phase, and control variables).1
• We causally validate these features through fine-grained steering: targeted changes in visual perception, semantic representation, and spatial localization produce predictable changes in downstream action prediction.1
• Overall, our results show how matching featurizers to representation geometry can provide more effective tools for understanding and changing robot behavior.1

Sources (1)

• [1]Multi-Dimensional Featurizers Yield Fine-Grained Steering and Data Shaping in Robot Foundation Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 06:15 PM

  Can better feature discovery improve control over robot behavior through internal interventions and training data?

  Recent work in vision and language models reveals multidimensional features, motivating block-sparse featurizers (BSFs), which represent each feature in a low-dimensional subspace.

Extractive summary: sentences quoted from the sources.