Where Do LLM Ensemble Gains Come From? Decomposing Candidate Value, Diversity, and Aggregation
LLM ensemble gains are often measured against a single baseline, but the value attributed to additional candidates can change with the comparison used.
ProofPaper ↗
Key points
- We study an LLM aggregator under four matched conditions: Direct (no candidates), Single-slot (one candidate in a multi-slot interface), Duplicate (one candidate repeated), and Multi-candidate (separately generated candidates).
- Under either non-diverse control, overall candidate gain (CIV) splits algebraically into an additional-candidate contrast (DSV) and a candidate-context contrast (CCV); the split, but not CIV, depends on the chosen control.
- In FRAMES evidence aggregation, candidate-context contrasts remain strongly negative under both non-diverse controls.
- Our results show why ensemble evaluations should state the reference control before attributing gains to additional candidates.
Sources (1)
- [1]Where Do LLM Ensemble Gains Come From? Decomposing Candidate Value, Diversity, and AggregationarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 05:36 PM
LLM ensemble gains are often measured against a single baseline, but the value attributed to additional candidates can change with the comparison used.
We study an LLM aggregator under four matched conditions: Direct (no candidates), Single-slot (one candidate in a multi-slot interface), Duplicate (one candidate repeated), and Multi-candidate (separately generated candidates).
Extractive summary: sentences quoted from the sources.