Auditing Tool Traces in Agentic LLMs: Source Localization and Counterfactual Dependence Testing
Agentic LLMs increasingly support scientific prediction by grounding decisions in external evidence.
ProofPaper ↗
Key points
- Stage 1, Per-Tool TF-IDF Decomposition (PT2D), localizes label-predictive patterns to individual tool results while avoiding signal dilution in the concatenated tool trace.
- Stage 2, Token-Class Redaction, counterfactually tests whether the frozen LLM depends on the localized shortcut.
- We apply this audit to TCR-pMHC binding, where tool augmentation substantially outperforms a no-evidence baseline.
- These findings show how hidden label shortcuts can masquerade as scientific reasoning or discovery, motivating source-level audits before benchmark gains are trusted.
Sources (1)
- [1]Auditing Tool Traces in Agentic LLMs: Source Localization and Counterfactual Dependence TestingarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 18, 09:17 PM
Agentic LLMs increasingly support scientific prediction by grounding decisions in external evidence.
Stage 1, Per-Tool TF-IDF Decomposition (PT2D), localizes label-predictive patterns to individual tool results while avoiding signal dilution in the concatenated tool trace.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 17, 2026EdgeKernelBench: Unified LLM-Driven Kernel Generation for Edge Backends
- Oct 16, 2026[Research] 2,788 tokens/s on a single RTX 5090 with a diffusion LLM — Introducing ORL (fully open source)
- Oct 16, 2026Characterising and Analysing Retrieval-Augmented Generation Techniques
- Oct 16, 2026Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline
- Oct 16, 2026OR-License: Evaluating LLM Agents as Real-World Operations Research Practitioners
- Oct 16, 2026Language Models Learn to Improve Language Models