Stronger Judges Do Not See More: Deployment-Integrity Blindness in LLM Evaluation
We introduce DIB-Bench, a benchmark that measures this deployment-integrity blindness across 7,560 scored examples, six tamper types, and an evaluator ladder from regex scorers through LLM judges to GPT-5-nano.
ProofPaper ↗
Key points
- LLM-agent evaluation reports behavioral scores but leaves the identity of the deployed runtime stack unreported.
- A tampered deployment that preserves evaluated outputs passes every output-only evaluator.
- Stronger evaluators accept more tampered outputs: mean acceptance rises from 88.6% (rule-based) to 94.1% (GPT-5-nano), and a trigger-conditional tamper invisible on the evaluation suite produces 48-100% harmful outputs, rising with model capability.
- We also propose a graded deployment-identity descriptor for evaluation reports.
Sources (1)
- [1]Stronger Judges Do Not See More: Deployment-Integrity Blindness in LLM EvaluationarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 02:44 AM
We introduce DIB-Bench, a benchmark that measures this deployment-integrity blindness across 7,560 scored examples, six tamper types, and an evaluator ladder from regex scorers through LLM judges to GPT-5-nano.
LLM-agent evaluation reports behavioral scores but leaves the identity of the deployed runtime stack unreported.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 15, 2026Beyond Compilation: Evaluating Faithful Natural-Language-to-Lean Statement Formalization
- Oct 15, 2026GlobeBench: Evaluating Text-Based World Model Capabilities for Scalable Environment Simulations
- Oct 15, 2026Librarian Agents: Training LLMs to Curate Long-Context Workspaces that Anticipate Future Queries
- Oct 15, 2026When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill
- Oct 14, 2026From Failures to Learnable Worlds: Policy-Conditioned Environment Generation for Navigation Agents
- Oct 13, 2026Plan or Retrieve First? A Scale Study of Tool Discovery Architectures