The Format is the Filter: Scoring Format Fixes What an LLM Judge Can Express
When an LLM judge filters instruction-tuning data, its scoring format (binary verdict, 1--5 rating, 0--100 score, or comparison) is rarely justified or ablated.
ProofPaper ↗
Key points
- Holding the candidate pool, judge prompt, decoding and selection budget fixed, we score 15,406 UltraFeedback candidates under five formats with three judges, curate, fine-tune 66 models, and evaluate them with four judges.
- Kendall's $τb$ hides them: it ranks binary best while $τa$ and top-1 agreement rank it last, because it discounts the ties a selector must break at random.
- Both comparative formats inherit a slot bias whose sign differs across judges.
- We propose four quantities that judge-based curation pipelines should report.
Sources (1)
- [1]The Format is the Filter: Scoring Format Fixes What an LLM Judge Can ExpressarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 05:36 PM
When an LLM judge filters instruction-tuning data, its scoring format (binary verdict, 1--5 rating, 0--100 score, or comparison) is rarely justified or ablated.
Holding the candidate pool, judge prompt, decoding and selection budget fixed, we score 15,406 UltraFeedback candidates under five formats with three judges, curate, fine-tune 66 models, and evaluate them with four judges.
Extractive summary: sentences quoted from the sources.