Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy

Key points

• LLM agents that compute the right answer can still answer wrongly.1
• When agents read the simulator's report through a section index rather than in context, GPT-5.1 and GPT-5.2 lose 11.3 and 7.9 points of accuracy, while Claude Opus 4.6 and Claude Sonnet 4.6 show no detectable cost.1
• Naming the authoritative output raises accuracy by 6.3-7.9 points over plain retrieval for the three GPT models; for GPT-5.1 and GPT-5.2, inlining outputs that fit works as well, while asking for correct rounding does not.1
• Context-management choices in agent harnesses should be judged by where accuracy is lost, not only by final accuracy.1

Sources (1)

• [1]Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 05:57 PM

  We ask where this costs accuracy with a real scientific tool: PHREEQC, a widely used geochemistry simulator from the USGS, whose reports reach 846.8k characters and whose determinism lets every printed value be traced.

  LLM agents that compute the right answer can still answer wrongly.

Extractive summary: sentences quoted from the sources.