Commit or Abstain? System One Decision Models, LLMs, and Missing Legal Authority

Key points

• Given the enormous number of claims to be checked, verifiable legal AI requires scalable, cost-effective models that answer the practical needs of law.1
• We test whether such models abstain when the correct authority is missing.1
• Accepting that LLM's answer only when a decision model agrees halves its wrong citations, and distilling a decision model's probabilities into Qwen3-8B, without labels, yields a well-calibrated local model.1
• Legal AI should be judged by its wrong citations, not by accuracy alone.1

Sources (1)

• [1]Commit or Abstain? System One Decision Models, LLMs, and Missing Legal Authority
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 04:17 PM

  We extend CaseHOLD with an abstention condition in which the correct holding is replaced in 1,670 of 5,314 items, and compare four decision models with six generative LLMs. When no option is correct, generative LLMs still pick one in 38-86% of items; the most cautious decision model does so in 30%.

  Given the enormous number of claims to be checked, verifiable legal AI requires scalable, cost-effective models that answer the practical needs of law.

Extractive summary: sentences quoted from the sources.