Beyond Compilation: Evaluating Faithful Natural-Language-to-Lean Statement Formalization
We measure how far compilation overstates faithfulness in statement autoformalization without canonical Lean targets.
ProofPaper ↗
Key points
- Lean verifies that a generated declaration is well typed, but cannot check whether it states what the user intended.
- Our criterion requires Lean compilation and agreement of two LLM judges from different model families.
- Across eight systems on 227 university-level statements, the gap between compiling and accepted outputs ranges from 1.3 to 29.5 percentage points: our tool-augmented agent compiles 87.2% of the statements, but only 57.7% are accepted.
- When GPT-5.2, one of the two judges, grades the translations of two models released after it, GPT-6 Astra and Claude Opus 5.5, it accepts 83% of the statements for each, against 21% for one-shot GPT-5.2.
Sources (1)
- [1]Beyond Compilation: Evaluating Faithful Natural-Language-to-Lean Statement FormalizationHugging Face Daily Papers · Oct 15, 12:00 AM
We measure how far compilation overstates faithfulness in statement autoformalization without canonical Lean targets.
Lean verifies that a generated declaration is well typed, but cannot check whether it states what the user intended.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 14, 2026CausalEvo: Causal Skill Self-Evolution for Health LLM Agents
- Oct 14, 2026ARIA: Runtime Inspection and Amendment for Multi-Agent Systems
- Oct 14, 2026INFUSE: A Unified Surface Benchmark for Indirect Prompt Injection in Agentic Systems
- Oct 14, 2026Beyond LLM Judge: Mechanical Verification for Agentic Autoformalization
- Oct 14, 2026How Third Moment Research runs 10,000 agents for 36+ hours
- Oct 13, 2026Gemini 4 Argon: Built to solve your hardest problems