PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviews

Key points

• Peer review is under strain from the growth of submissions, and review comments written by humans or AI can misstate what a paper says.1
• A frontier LLM checks comments well but must read the whole paper for each check, while cheap decision models, which score whether a statement holds in a text, fall far short on whole comments that mix several implicit assertions.1
• PremiseCheck is a framework that combines the two.1
• When each comment is checked on its own, PremiseCheck reaches accuracy comparable to GPT-5.5 across all three fields at about half its cost, and on NLP reviews at a quarter of its cost and less than half its latency.1

Sources (2)

• [1]PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviews
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 12:42 AM

  We introduce ReviewFact, a benchmark that labels such factual errors in review comments under one definition across NLP, machine learning, and the natural sciences, and that measures cost and latency alongside accuracy.

  Peer review is under strain from the growth of submissions, and review comments written by humans or AI can misstate what a paper says.

• [2]PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviews
  Hugging Face Daily Papers · Oct 15, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.