LegalRewardBench: Benchmarking Reward Models for Legal Preference Evaluation

Key points

• Reward models are central to preference alignment for legal tasks, yet existing reward model benchmarks are still primarily centered on general-domain preferences.1
• The benchmark organizes legal preference evaluation into five capability levels: knowledge, understanding, analysis, decision, and writing, and preference examples are constructed through rubric-guided automatic judging combined with human verification.1
• We evaluate three families of judges, covering both general and fine-tuned models, and generative and discriminative architectures.1
• Experimental results show that strong general-purpose generative models are currently the most robust legal reward models, while existing fine-tuned reward models transfer less reliably to legal preference evaluation.1

Sources (1)

• [1]LegalRewardBench: Benchmarking Reward Models for Legal Preference Evaluation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 12:47 PM

  In this work, we introduce LegalRewardBench, the first human-verified benchmark for evaluating reward models in the legal domain.

  Reward models are central to preference alignment for legal tasks, yet existing reward model benchmarks are still primarily centered on general-domain preferences.

Extractive summary: sentences quoted from the sources.