ResearchResearch paperEfficiency & Inference · Large Language Models · Robotics & Embodied AI1 source · Oct 14, 2026

Beyond LLM Judge: Mechanical Verification for Agentic Autoformalization

We introduce a novel method for verifying Lean formalizations that goes beyond LLM-judged faithfulness.

Unlike most verification methods, which rely on LLMs or trained models, our method combines LLM-based semantic review with mechanical numerical and vacuity checks, providing checkable feedback and detecting errors missed by LLM judges.
We apply this method, along with our Extract-Verify-Proof autoformalization pipeline, to construct the Formalized Library of Mathematical Functions (FLMF), a partial formalization of the NIST Digital Library of Mathematical Functions (DLMF).
On three labeled datasets, our method achieves the lowest accepted-error rate among the evaluated methods, measured as the fraction of incorrect statements accepted by the verification method.
Expert review of the results also identifies translation and source defects in existing datasets.

[1]Beyond LLM Judge: Mechanical Verification for Agentic Autoformalization
arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 05:54 PM
We introduce a novel method for verifying Lean formalizations that goes beyond LLM-judged faithfulness.
Unlike most verification methods, which rely on LLMs or trained models, our method combines LLM-based semantic review with mechanical numerical and vacuity checks, providing checkable feedback and detecting errors missed by LLM judges.

Extractive summary: sentences quoted from the sources.

Before this