Write Once, Check Many: Reusable Ledgers for Post-Hoc Policy Compliance Auditing of Agent Trajectories

Key points

• Auditing LLM agents against policies is critical to ensure safety but requires rereading long trajectories once per policy.1
• Can we instead record each trajectory once and reuse that record for policies not known in advance?1
• On 900 trajectories from SHADE-Arena, $τ$-bench, and $τ^2$-bench, we show that a 12B reader of narrative ledgers can either match or in fact achieve higher AUC than the same 12B model reading full trajectories.1
• Our results suggest that reusable, policy-independent records can make retrospective agent auditing more efficient and effective, and highlight important considerations in ledger design.1

Sources (1)

• [1]Write Once, Check Many: Reusable Ledgers for Post-Hoc Policy Compliance Auditing of Agent Trajectories
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 17, 06:04 AM

  We propose write once, check many, a framework in which a writer model converts each agent trajectory into a policy-independent ledger of the agent's actions and outcomes.

  Auditing LLM agents against policies is critical to ensure safety but requires rereading long trajectories once per policy.

Extractive summary: sentences quoted from the sources.