Beyond Scalar Leaderboards: Adaptive Sampling for Pareto Frontier Identification in Multi-Objective LLM Arenas
In this paper, we present Vector-Borda Empirical-Gap Elimination (VB-EGE), a fixed-confidence adaptive sampling algorithm for efficiently identifying models on the Pareto frontier in arena-style evaluation, where each comparison evaluates a pair of models on a prompt with respect to a selected performance dimension.
ProofPaper ↗
Key points
- The evaluation of large language models (LLMs) is inherently multi-objective, as different models may offer different tradeoffs across performance dimensions such as math, code, and writing.
- VB-EGE progressively eliminates models from an active set once sufficient statistical evidence has been accumulated to certify their Pareto status.
- Under a multi-objective Bradley-Terry model, we show that VB-EGE returns the correct Pareto frontier with high probability and derive an instance-dependent upper bound on its sample complexity.
- Simulations and experiments on three real-world Arena datasets (Arena, SearchArena, and VisionArena) demonstrate VB-EGE's empirical effectiveness.
Sources (1)
- [1]Beyond Scalar Leaderboards: Adaptive Sampling for Pareto Frontier Identification in Multi-Objective LLM ArenasarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 12:50 AM
In this paper, we present Vector-Borda Empirical-Gap Elimination (VB-EGE), a fixed-confidence adaptive sampling algorithm for efficiently identifying models on the Pareto frontier in arena-style evaluation, where each comparison evaluates a pair of models on a prompt with respect to a selected performance dimension.
The evaluation of large language models (LLMs) is inherently multi-objective, as different models may offer different tradeoffs across performance dimensions such as math, code, and writing.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 20, 2026MPS in PyTorch 2.15
- Oct 19, 2026Evaluating Agents on Highly Subjective Tasks
- Oct 19, 2026Introducing Clairic, a new System One class of extraction models
- Oct 19, 2026Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM
- Oct 19, 2026From Black Box to Controller: Inference-Time Steering of LLM via Feature Amplification
- Oct 18, 2026Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark: DeepSeek-V4.1-Flash-1017 EXL3 2.9bpw on 2x NVIDIA DGX Spark (GB10) with TensorFold: 2.1-2.3x the vLLM kit's decode speed.
