Adaptation of Computer Use Agents to New Software with User Manuals

Key points

• Recent advances in multimodal foundation models have significantly improved the ability to operate real-world software through graphical user interfaces (GUIs) using visual observations and natural language instructions.1
• During execution, MAGE retrieves task-relevant manual content and uses a planner to generate explicit procedural guidance for CUAs.1
• To further account for the discrepancies between manuals and actual GUI environments, such as version differences or user settings, MAGE also adapts the agent policy through environment interaction.1
• Experimental results across six applications demonstrate improvements in task success rates over strong baselines, highlighting the effectiveness of manual-guided planning and policy adaptation in new software environments.1

Sources (1)

• [1]Adaptation of Computer Use Agents to New Software with User Manuals
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 06:44 AM

  In this paper, we propose MAGE (Manual-augmented computer use AGent Execution and learning), a framework that leverages user manuals as a scalable source of procedural knowledge to improve both inference and training.

  Recent advances in multimodal foundation models have significantly improved the ability to operate real-world software through graphical user interfaces (GUIs) using visual observations and natural language instructions.

Extractive summary: sentences quoted from the sources.