Measuring the Task Value of Precision Upgrades in Closed-Loop Agents

Key points

• A learned precision router chooses where to call it from the disagreement between the two models and is judged by end-to-end success.1
• An inserted BF16 action, however, can rescue a failing task or damage a succeeding one.1
• On Qwen3 agents with simulated 4-bit weights in tau2-bench, replacing the first action where the two models differed in tool use changed many task outcomes, but rescues and damages nearly cancelled, and on held-out telecom tasks the action-only estimate was negative.1
• Two references changed outcomes at fixed bit width: replacing an action with that of a second 4-bit checkpoint changed about as many outcomes as replacing it with the BF16 action, and reordering tool-argument keys changed outcomes while the executed tool call stayed identical.1

Sources (1)

• [1]Measuring the Task Value of Precision Upgrades in Closed-Loop Agents
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 17, 12:29 AM

  A low-precision agent can call its full-precision (BF16) model at selected decisions as a fallback.

  A learned precision router chooses where to call it from the disagreement between the two models and is judged by end-to-end success.

Extractive summary: sentences quoted from the sources.