SoK: Grading Open-Text Legal Answers, from Law School Exams to Trustworthy Rubric Judges
This paper systematizes how open-text legal answers can be evaluated.
ProofPaper ↗
Key points
- Law students are examined through open-text answers: case solutions, essays and memos that apply the law to a set of facts.
- Such answers are expensive to grade, human graders disagree widely on them, and large language models (LLMs) now offer to grade them at scale, both in teaching and in the benchmarks that test legal AI.
- We then compare eight evaluation paradigms against them, from unstructured human grading to LLM judges with instance-specific rubrics and pairwise comparison.
- From this we derive the trustworthy rubric judge: an instance rubric that credits arguments and alternative paths,evidence checked in code, probes, repeated runs, calibration against independent human grades,logging, and an examiner who reads every script.
Sources (1)
- [1]SoK: Grading Open-Text Legal Answers, from Law School Exams to Trustworthy Rubric JudgesarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 11:09 AM
This paper systematizes how open-text legal answers can be evaluated.
Law students are examined through open-text answers: case solutions, essays and memos that apply the law to a set of facts.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 14, 2026PN-Mix: Pair-Norm Guided Mixed-Precision Key Cache Quantization for Efficient Long-Output LLM Inference
- Oct 14, 2026RuleTrek: Benchmarking Agentic Rule Reasoning in Complex Scenarios
- Oct 14, 2026SafeDriveVLA: Navigation-Conditioned World Model Dreaming for Conflict-Aware End-to-End Autonomous Driving
- Oct 14, 2026Sparse Autoencoders Are a Shared Coordinate System for Symbolic Matching
- Oct 13, 2026Plan or Retrieve First? A Scale Study of Tool Discovery Architectures
- Oct 13, 2026Beyond the Verdict: Agentic Code Judging with Verifiable Feedback