CME-SpectrumBench: Can LLMs Analyze Condensed Matter Spectral Data?

Key points

• Large Language Models (LLMs) are presently well assessed by expert-level mathematics and physics benchmarks which focus on theoretical, symbolic reasoning.1
• Using simulated CME spectral data, we prompt models to identify structures from 2D intensity arrays in the face of noise and limited instrumental resolution, a test of dense context reasoning.1
• LLM performance becomes comparable to handwritten code by allowing tool use (access to a Python interpreter) with/without ReAct prompting, in effect mapping questions onto agentic code generation problems.1
• Our findings show that frontier models can serve as research assistants by interfacing with data through code, but still face difficulties processing information directly from large datasets - a key barrier future foundation models must overcome.1

Sources (1)

• [1]CME-SpectrumBench: Can LLMs Analyze Condensed Matter Spectral Data?
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 10:58 PM

  To this end, we present CME-SpectrumBench, an expert-designed benchmark consisting of 190 questions that measures how reliably frontier LLMs analyze research-level spectral scientific data in condensed matter experiments (CME).

  Large Language Models (LLMs) are presently well assessed by expert-level mathematics and physics benchmarks which focus on theoretical, symbolic reasoning.

Extractive summary: sentences quoted from the sources.