Rubrics Carry the Standard: Shapley Attribution for Checklist-Based LLM Judges

Key points

• Two assumptions surround this practice: that leniency or strictness is a property of the judge model, and that a rubric can be improved by asking a model to review it.1
• We test both in a controlled grading task with planted flaws and on two human-labeled datasets, SummEdits and RAGTruth.1
• Holding the judge and the outputs fixed, rubric content moves the pass rate of correct outputs from 2% to 100%, whereas changing the judge model or the rendering of the same checks moves it by a few points.1
• To measure and control this, we show that under all-must-pass aggregation the Shapley value of each check has an exact closed form, which yields per-check catch credit, false-fail blame and unique catches in linear time.1

Sources (1)

• [1]Rubrics Carry the Standard: Shapley Attribution for Checklist-Based LLM Judges
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 18, 03:01 AM

  Checklist rubrics applied by large language model (LLM) judges have become a standard instrument for evaluating open-ended outputs and for rewarding them during training.

  Two assumptions surround this practice: that leniency or strictness is a property of the judge model, and that a rubric can be improved by asking a model to review it.

Extractive summary: sentences quoted from the sources.