Selecting the Best Few with Human Evaluators: Adaptively Combining Ratings and Pairwise Comparisons
This paper considers a decision-maker tasked with identifying the $k$ best items from a large pool of candidates, such as job applicants, LLMs, award nominees, or new product ideas.
ProofPaper ↗
Key points
- The decision-maker learns about the items' quality by soliciting human evaluations.
- Each evaluation can be costly, so the decision-maker aims to use the limited evaluation budget efficiently by allocating evaluations adaptively.
- Two popular forms are ratings and pairwise comparisons, and we allow for both, as they may be especially useful in combination given the limited budget.
- To solve the allocation problem, we develop an adaptive algorithm built from two allocation policies.
Sources (1)
- [1]Selecting the Best Few with Human Evaluators: Adaptively Combining Ratings and Pairwise ComparisonsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 01:32 PM
This paper considers a decision-maker tasked with identifying the $k$ best items from a large pool of candidates, such as job applicants, LLMs, award nominees, or new product ideas.
The decision-maker learns about the items' quality by soliciting human evaluations.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 19, 2026AdaExploreBench: Benchmarking LLM Agents for Proactive Exploration and Test-time Adaptation in Evolving Environments
- Oct 19, 2026Unsupervised Tool Evolution for Coding Agents in Data-Lake Analytics
- Oct 19, 2026TokenGate: Adapting LLMs for Fast and Memory-Efficient Prefill with Learned Token Pruning
- Oct 17, 2026EdgeKernelBench: Unified LLM-Driven Kernel Generation for Edge Backends
- Oct 17, 2026Axiomatic World Modeling for Physics Reasoning
- Oct 16, 2026[Research] 2,788 tokens/s on a single RTX 5090 with a diffusion LLM — Introducing ORL (fully open source)