Benchmark for Structured Retrieval-Augmented Generation in Perovskite Photovoltaics
While retrieval-augmented language models increasingly serve as literature assistants in the physical sciences, existing benchmarks evaluate them primarily on qualitative text overlap or through model-based scoring rather than against verified experimental ground truth.
ProofPaper ↗
Key points
- In experimental materials disciplines such as perovskite photovoltaics, however, the most critical inquiries are inherently quantitative and aggregate measurements across dozens or hundreds of papers.
- Here, we present a quantitative question-answering benchmark where all 755 ground-truth answers are computed directly from 45,441 measured device records (40,889 human-curated records through 2021 and 4,552 records extracted from 2022-2026 literature).
- Benchmarking six retrieval architectures over 158,274 passages from 25,662 papers using an open 14B model on a single 24 GB GPU, standard passage retrieval tops out at 16-23% accuracy (compared to 1.6-6.5% zero-shot).
- Aggregate queries spanning extrema, distributions, and filtered record counts prove virtually unsolvable via text retrieval alone, demonstrating that systematic table extraction is an operational prerequisite for literature-scale quantitative inquiry.
Sources (1)
- [1]Benchmark for Structured Retrieval-Augmented Generation in Perovskite PhotovoltaicsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 18, 10:37 PM
While retrieval-augmented language models increasingly serve as literature assistants in the physical sciences, existing benchmarks evaluate them primarily on qualitative text overlap or through model-based scoring rather than against verified experimental ground truth.
In experimental materials disciplines such as perovskite photovoltaics, however, the most critical inquiries are inherently quantitative and aggregate measurements across dozens or hundreds of papers.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 18, 2026Aligning Language Models with Indigenous Community Cosmovision through Constitutional AI
- Oct 18, 2026Meet Voxel51 3.0: A Unified Multimodal Data Platform for Physical AI
- Oct 16, 2026[Research] 2,788 tokens/s on a single RTX 5090 with a diffusion LLM — Introducing ORL (fully open source)
- Oct 16, 2026Enhancing RAG-based Medical VQA with Gated Multimodal Representations
- Oct 16, 2026Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline
- Oct 16, 2026Language Models Learn to Improve Language Models