MathVis: Can AI Reason through Drawing? A Benchmark for Mathematical Figure Generation

Key points

• The benchmark consists of 512 tasks, each comprising a mathematical statement, a reference diagram, executable TikZ or SVG code, a Gold Intermediate Representation (IR), atomic rubrics, and error codes.1
• Through rigorous evaluation, GPT-6-Astra, Claude Fable 5, and Gemini 3.1 Pro Preview achieve pass@1 scores of 31.25%, 26.56%, and 22.85%, respectively.1
• We found that although the tasks require visualization, LLMs produce outputs dominated by formal languages, symbols, and graphics programs, rather than explicit diagram-mediated reasoning.1
• Ultimately, this benchmark aims to visualize mathematical problems and promote research on enabling LLMs to simplify their reasoning via visual diagrams.1

Sources (1)

• [1]MathVis: Can AI Reason through Drawing? A Benchmark for Mathematical Figure Generation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 03:52 AM

  This paper introduces MathVis, a benchmark designed to evaluate LLMs' ability to visualize frontier mathematical problems and perform diagram-assisted reasoning.

  The benchmark consists of 512 tasks, each comprising a mathematical statement, a reference diagram, executable TikZ or SVG code, a Gold Intermediate Representation (IR), atomic rubrics, and error codes.

Extractive summary: sentences quoted from the sources.