Judging the Judges: Confidence and Disagreement in Rubrics as Rewards

Key points

• We first show that judge identity alone produces a substantial spread in scores when training Qwen3-4B-Base on RaR-Medicine and RaR-Science across five frontier and open-weight judges.1
• Building on this, we propose RaRE (Rubrics as Rewards with Ensembles), a novel judge ensemble strategy for RaR with two mechanisms: a soft-Likert reward that uses judge logprobs to weight more confident judges more heavily, and a drop-on-disagreement mask that excludes high-disagreement rollouts from the GRPO update.1
• We find that this design is critical: while a naive average of judge scores fails to outperform the best single judge, RaRE outperforms every single-judge RaR variant on both HealthBench and GPQA Diamond, with per-theme analysis showing that it inherits the complementary strengths of its constituent judges.1
• Our findings establish judge ensembling as a compelling axis for improving RaR.1

Sources (1)

• [1]Judging the Judges: Confidence and Disagreement in Rubrics as Rewards
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 03:35 AM

  Rubrics as Rewards (RaR) extends reinforcement learning to non-verifiable domains by using an LLM judge to score responses against a structured rubric.

  We first show that judge identity alone produces a substantial spread in scores when training Qwen3-4B-Base on RaR-Medicine and RaR-Science across five frontier and open-weight judges.

Extractive summary: sentences quoted from the sources.