A Mechanistic Study of Automated Failure Attribution in LLM-based Agentic Systems

Key points

• However, this collaborative structure introduces new challenges for reliably evaluating and diagnosing system failures, particularly failure attribution: tracing a system's execution to identify the agent step responsible for an incorrect final outcome.1
• Using linear probes applied to internal activations, we uncover a significant knows-but-won't-say gap: models encode sufficient information to attribute a failure correctly but fail to explicitly output the responsible step.1
• Building on this finding, we propose grafting, a simple manual intervention that exploits knowledge already encoded in the model to improve AFA performance without requiring ground-truth labels at inference time.1
• Extensive evaluations across multiple LLMs and benchmark settings demonstrate that grafting consistently outperforms direct-prompting approaches, underscoring the potential of leveraging latent model knowledge for more reliable failure attribution.1

Sources (1)

• [1]A Mechanistic Study of Automated Failure Attribution in LLM-based Agentic Systems
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 06:33 PM

  Agentic systems have demonstrated strong capabilities on complex tasks by enabling multiple interacting large language model (LLM)-based agents to collaborate and execute subtasks in parallel.

  However, this collaborative structure introduces new challenges for reliably evaluating and diagnosing system failures, particularly failure attribution: tracing a system's execution to identify the agent step responsible for an incorrect final outcome.

Extractive summary: sentences quoted from the sources.