Work2Skill: Capturing Human GUI Experience for Reusable Agent Skill Induction

Key points

• Computer-use agents can perform increasingly complex GUI tasks, yet unfamiliar software and workflows requiring human domain knowledge remain challenging.1
• Human demonstrations convey procedural knowledge through the same GUI interaction space, but reuse requires interpreting actions in context and separating workflow rules from task-specific details.1
• WorkRecorder provides the framework's infrastructure for collecting this experience, aligning screen recordings, mouse and keyboard events, and available accessibility-tree information.1
• We then propose an agentic skill-induction method that inspects evidence on demand to produce reusable skills capturing workflow rules, step dependencies, and completion checks.1

Sources (2)

• [1]Work2Skill: Capturing Human GUI Experience for Reusable Agent Skill Induction
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 04:32 AM

  We thus introduce Work2Skill, a framework that captures human GUI experience for reusable agent skill induction.

  Computer-use agents can perform increasingly complex GUI tasks, yet unfamiliar software and workflows requiring human domain knowledge remain challenging.

• [2]Work2Skill: Capturing Human GUI Experience for Reusable Agent Skill Induction
  Hugging Face Daily Papers · Oct 9, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.