Annotrace: A Workbench for Step-Level Annotation of Agent Traces

Key points

• Every study of why LLM agents fail begins by labelling traces by hand, and every such study builds its own throwaway tool.1
• The annotator writes their own labelling codebook; a frontier model proposes a starting set from a sample of their own traces, answers first, and is asked exactly the questions the human is asked, making annotation a joint human-AI process.1
• Annotrace measures agreement between any two annotators, human or AI.1
• On a multi-agent question-answering corpus it reproduces a previous manual effort's 235 annotation units exactly, and two models labelling the same traces agree 91% of the time at $κ= 0.82$.1

Sources (1)

• [1]Annotrace: A Workbench for Step-Level Annotation of Agent Traces
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 10:55 AM

  Observability platforms record agent traces but offer no structured labelling; annotation tools label text but have no notion of a step, a delegation or a tool call, leaving it unclear what the smallest unit of execution to label should be.

  Every study of why LLM agents fail begins by labelling traces by hand, and every such study builds its own throwaway tool.

Extractive summary: sentences quoted from the sources.