Are We Testing Too Much? A Survey of Benchmark Compression for Efficient Evaluation of Large Language Models
Benchmarks are essential for evaluating and improving large language models (LLMs).
ProofPaper ↗
Key points
- Yet many widely used benchmarks are redundant, making evaluation unnecessarily expensive.
- Benchmark compression lowers this cost by estimating full-benchmark scores and rankings from a partial evaluation.
- We review 119 papers to explain how methods choose what to evaluate, recover full-benchmark results, and verify their reliability.
- From this analysis, we develop a taxonomy covering compression within individual benchmarks, across benchmark suites, and during adaptive evaluation, together with work on redundancy and reliability.
Sources (2)
- [1]Are We Testing Too Much? A Survey of Benchmark Compression for Efficient Evaluation of Large Language ModelsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 12:15 PM
Benchmarks are essential for evaluating and improving large language models (LLMs).
Yet many widely used benchmarks are redundant, making evaluation unnecessarily expensive.
- [2]Are We Testing Too Much? A Survey of Benchmark Compression for Efficient Evaluation of Large Language ModelsHugging Face Daily Papers · Oct 13, 12:00 AM · same content
Extractive summary: sentences quoted from the sources.
Before this
- Oct 12, 2026Meet the Anthropic Engineer Keeping Its AI Agent Trustworthy
- Oct 12, 2026Declarative Reactive Robot Behaviors from Imperative Language Models
- Oct 12, 2026AI Representations of Mental Health Stigma: Quantifying the Gap Between AI Estimations and Human Perceptions Across Gender Groups Using GPT-Based LLMs
- Oct 12, 2026Spectral-Driven Adaptive-Rank Optimizer for Large Language Model Reinforcement Learning with Verifiable Rewards
- Oct 12, 2026Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged
- Oct 12, 2026Rethinking LLM-Guided Visual Question Answering: Does Deeper Knowledge Integration Really Help?