Auditing Failure Attribution: Label Predictability and Judge Behavior
Failure-attribution benchmarks ask an LLM judge which step or agent caused a failure.
ProofPaper ↗
Key points
- However, spurious signals (surface cues) that predict the annotation can produce a high score.
- In this paper, we propose four tests to separate checking from cue use.
- Residual accuracy scores judges where cue-only predictors fail.
- Adding task-related reasoning after that step moves this cue: fine-tuned judges lose 31-99 points, but zero-shot judges lose at most 9, motivating cue following.
Sources (1)
- [1]Auditing Failure Attribution: Label Predictability and Judge BehaviorarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 05:26 AM
Failure-attribution benchmarks ask an LLM judge which step or agent caused a failure.
However, spurious signals (surface cues) that predict the annotation can produce a high score.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 12, 2026DTA-GT: Direction- and Topology-Aware Graph Transformer for Neural Network Representation Learning
- Oct 12, 2026Not Every Cue Is Evidence: Cue Supportability for Forensic Distillation in Face Forgery Detection
- Oct 12, 2026RelLedger: Relational Information Governance for LLM Agent Organizations
- Oct 12, 2026AI Representations of Mental Health Stigma: Quantifying the Gap Between AI Estimations and Human Perceptions Across Gender Groups Using GPT-Based LLMs
- Oct 11, 2026RetryBench: Separating Retry Permission from Recovery Outcomes in GPT Tool Use
- Oct 11, 2026Stereo-Planner: Learning Stereo-to-Trajectory Planning for Zero-Shot Sim-to-Real Quadrotor Navigation