How Many Repeated Pairwise Comparisons Are Needed for Ranking under Heterogeneity?

Key points

• We investigate how many repeated comparisons within each user-task context are necessary and sufficient for ranking recovery.1
• Under a heterogeneous Bradley-Terry model with fixed inverse temperature, we start with a naive MLE-based algorithm that requires $Ω(1/Δ^2)$ repeated comparisons per context to ensure ranking recovery.1
• We then present two MLE-based variants and a randomized Russian Roulette-style algorithm that recover the ranking using $O(\log(1/Δ))$ repeated comparisons per context, and we prove that this logarithmic dependence is optimal.1
• Synthetic experiments and semi-synthetic experiments based on Arena data compare the four algorithms in settings with varying levels of preference heterogeneity and under varying context distributions.1

Sources (1)

• [1]How Many Repeated Pairwise Comparisons Are Needed for Ranking under Heterogeneity?
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 06:52 PM

  We study ranking models by population-average utility from pairwise comparisons when preferences vary across users and tasks.

  We investigate how many repeated comparisons within each user-task context are necessary and sufficient for ranking recovery.

Extractive summary: sentences quoted from the sources.