Lost in Recovery: Demystifying Recovery Behavior of Computer-Use Agents via Goal-Oriented Action Replay

Key points

• Recent advances in large (vision) language models have enabled computer-use agents that automate digital workflows via fine-grained keyboard and mouse operations.1
• We formalize Transition-Consistent Recovery: identical downstream actions from the recovered and target states must yield identical outcomes.1
• We operationalize this criterion through goal-oriented action replay following controlled error injection.1
• Using ReTrace, we present the first systematic study of autonomous, model-based recovery, revealing that (i) ideal recovery can unlock model's latent task completion capabilities, (ii) current computer-use agents fail to exploit their latent recovery capabilities, and (iii) accurate model-based recovery yields measurable end-to-end gains.1

Sources (1)

• [1]Lost in Recovery: Demystifying Recovery Behavior of Computer-Use Agents via Goal-Oriented Action Replay
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 09:59 AM

  To address this gap, we present ReTrace, a scalable evaluation framework for systematically studying recovery behavior in computer-use agents.

  Recent advances in large (vision) language models have enabled computer-use agents that automate digital workflows via fine-grained keyboard and mouse operations.

Extractive summary: sentences quoted from the sources.