OptionWorld: A Benchmark for Diagnosing Failures of LLMs as Long-Term Personal Assistants

Key points

• Long-term assistants must follow unfinished tasks across days as information and circumstances change.1
• Finding information or starting an action, however, does not ensure that the task gets completed.1
• Failure analysis identifies the three most frequent problems: needed information is not obtained or combined, available opportunities to act go unused, and agents impose restrictions that the task does not require.1
• Three follow-up experiments show that supplying missing facts restores task success in some cases; action reminders can restart work while subsequent steps still miss the deadline; and correcting a supplied note that wrongly forbids contact restores permitted inquiry.1

Sources (1)

• [1]OptionWorld: A Benchmark for Diagnosing Failures of LLMs as Long-Term Personal Assistants
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 04:38 PM

  We introduce OptionWorld, a benchmark that traces information acquisition, judgments, actions, and outcomes in simulated daily-life tasks to identify where current LLMs fall short as long-term personal assistants.

  Long-term assistants must follow unfinished tasks across days as information and circumstances change.

Extractive summary: sentences quoted from the sources.