Good Judgment, Bad Advice: Decision-Model Controllers Raise Agent Cost Without Improving Success

Key points

• Agent scaffolds increasingly delegate small decisions (whether to stop, verify, or change approach) to cheaper models.1
• In a pre-specified study on Terminal-Bench 2.1, we hold one agent loop fixed and vary only the controller: none, deterministic rules, a small LLM (Claude Haiku 5.5) answering the same questions, Jev, and a placebo that samples Jev's development-phase choice frequencies at the same moments.1
• Jev was cheaper than the small-LLM controller (ratio 0.81) but 22% more expensive than the placebo, and no pass-rate difference between arms was detectable.1
• Offline, Jev ranked finish requests better than the alternatives (AUROC 0.78 vs. 0.62 for rules and 0.60 for the small LLM) but was poorly calibrated: its mean probability that a run was complete was 0.50 when 87% of those runs passed, so it asked the agent to verify work that was already correct.1

Sources (2)

• [1]Good Judgment, Bad Advice: Decision-Model Controllers Raise Agent Cost Without Improving Success
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 17, 11:53 AM

  We ask whether a typed decision model, Jev 1.13, consulted only at deterministic decision points, makes a Claude Sonnet 5.5 terminal agent cheaper without making it worse.

  Agent scaffolds increasingly delegate small decisions (whether to stop, verify, or change approach) to cheaper models.

• [2]Good Judgment, Bad Advice: Decision-Model Controllers Raise Agent Cost Without Improving Success
  Hugging Face Daily Papers · Oct 17, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.