Instructions Are Evidence, Not Objectives: Measuring Outcome Alignment of Agents

Key points

• In our analysis of 100 user-agent trajectories from SWE-chat, 56% of initial instructions omit requirements that later emerge as necessary, and 52% of the useful follow-up requirements could have been inferred from the project.1
• We call this ability outcome alignment.1
• In this work, we construct TiGanEval, a benchmark of software engineering and research tasks.1
• Evaluating eight frontier models including GPT-5.6-Sol, we find outcome alignment broadly missing: the models lose 53 to 67 points of success rate when moving from the full specification to the partial instruction, and consequence feedback across four submissions does not close the gap.1

Sources (1)

• [1]Instructions Are Evidence, Not Objectives: Measuring Outcome Alignment of Agents
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 05:39 AM

  Large language model agents can follow complex human instructions to solve real-world tasks.

  In our analysis of 100 user-agent trajectories from SWE-chat, 56% of initial instructions omit requirements that later emerge as necessary, and 52% of the useful follow-up requirements could have been inferred from the project.

Extractive summary: sentences quoted from the sources.