Multimodal Embodied Agents: A Unified Capability-centric Perspective from Computer-Use to Robot-Use

Key points

• We define multimodal embodied agents (MMEAs) as goal-directed systems that couple multimodal task reasoning with physical action and revise decisions from the resulting feedback.1
• This raises a central question: what changes when a multimodal agent moves from computer-use to robot-use?1
• We introduce PAPAV, a unified capability-centric framework of five recurring functions: Perceive the current state, Anticipate action effects, Plan a feasible course, Act through an interface or body, and Verify the outcome.1
• Five physical constraints, from partial observability to unverifiable outcomes, leave fewer choices fixed in advance and less room to reverse errors.1

Sources (2)

• [1]Multimodal Embodied Agents: A Unified Capability-centric Perspective from Computer-Use to Robot-Use
  Hugging Face Daily Papers · Oct 21, 12:00 AM

  Multimodal agents (MMAs) sustain interaction through reasoning, memory, tools, and feedback, most visibly as computer-use agents, while robotic systems (RSs) couple sensing and actuation under physical dynamics.

  We define multimodal embodied agents (MMEAs) as goal-directed systems that couple multimodal task reasoning with physical action and revise decisions from the resulting feedback.

• [2]Multimodal Embodied Agents: A Unified Capability-centric Perspective from Computer-Use to Robot-Use
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 05:52 PM

Extractive summary: sentences quoted from the sources.