StepBench: Diagnosing Real-World Procedural Understanding in AI Agents

Key points

• Evaluating an agent's planning ability remains challenging for open-domain tasks, where verifying procedures through execution is often impractical.1
• We additionally evaluate full-plan generation to examine how diagnostic performance correlates with end-to-end plan validity.1
• For generation, STEPEVAL identifies critical failures that prevent goal completion.1
• Across 14 configurations of 12 models, we find substantial differences across planning capabilities, with step ordering proving the most difficult: the strongest model correctly orders only 63.0% of procedures, dropping to 33.1% for those with nine or more steps.1

Sources (1)

• [1]StepBench: Diagnosing Real-World Procedural Understanding in AI Agents
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 04:27 PM

  We introduce STEPBENCH, a benchmark of 6,885 verified procedures crawled from wikiHow across 81 open-domain categories.

  Evaluating an agent's planning ability remains challenging for open-domain tasks, where verifying procedures through execution is often impractical.

Extractive summary: sentences quoted from the sources.