From VLM Planners to Robotic Agents: Self-Evolving Harnesses for History-Dependent Manipulation

Key points

• Existing work has begun to address these challenges by augmenting foundation-model policies with memory and by introducing hierarchical architectures that separate high-level planning from low-level execution.1
• We organize the high-level VLM as a robotic agent through an editable harness to maintain task context and improve planning through execution experience.1
• We instantiate this approach as SERA (Self-Evolving Robotic Agent), with a fixed VLA executor providing low-level control.1
• Our exploration suggests that agentic planning is a promising direction for history-dependent robotic manipulation.1

Sources (2)

• [1]From VLM Planners to Robotic Agents: Self-Evolving Harnesses for History-Dependent Manipulation
  Hugging Face Daily Papers · Oct 21, 12:00 AM

  Vision-language-action (VLA) models achieve strong performance across embodied manipulation tasks by unifying visual understanding, language conditioning, and action generation.

  Existing work has begun to address these challenges by augmenting foundation-model policies with memory and by introducing hierarchical architectures that separate high-level planning from low-level execution.

• [2]From VLM Planners to Robotic Agents: Self-Evolving Harnesses for History-Dependent Manipulation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 10:49 AM · same content

Extractive summary: sentences quoted from the sources.