Two Sample Sizes in Test-Time LLM Ensembling: Buffered ERM and Minimax Theory
Test-time scaling can improve the performance of large language models by generating multiple answers to a question and aggregating them.
ProofPaper ↗
Key points
- Learning these weights from a training dataset involves two sample sizes: $Q$ labeled questions and $N$ generations per model and question.
- Under a margin condition with exponent $β$, we show that empirical risk minimization (ERM) with a calibrated margin buffer achieves excess risk of order $Q^{-1/2}+N^{-β/2}$, up to logarithmic factors, for fixed numbers of models and candidate answers.
- When questions are relatively scarce, the $Q^{-1/2}$ term governs learning and the general buffered ERM is optimal.
- Questions remain about the transitions between regimes and the construction of matching pooled estimators for general ensembles.
Sources (1)
- [1]Two Sample Sizes in Test-Time LLM Ensembling: Buffered ERM and Minimax TheoryarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 06:41 AM
Test-time scaling can improve the performance of large language models by generating multiple answers to a question and aggregating them.
Learning these weights from a training dataset involves two sample sizes: $Q$ labeled questions and $N$ generations per model and question.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 15, 2026The Density of Thought: Predicting and Steering Reasoning in Large Language Models
- Oct 15, 2026TimeCap: A Scalable Open-Source Pipeline for Captioning Wearable Health Time-Series Data
- Oct 15, 2026Fractional Power-of-Two Quantization for Efficient and Effective Multiplier-Free LLM Inference
- Oct 15, 2026System1-Arena: Which Decisions Can Be Left to System 1?
- Oct 15, 2026Noise-Space Reward Alignment for Few-Step Generative Models
- Oct 15, 2026Librarian Agents: Training LLMs to Curate Long-Context Workspaces that Anticipate Future Queries