FinGREAT: Evaluating LLMs on Financial Graph-guided Retrieval And Typing
We investigate whether open-weight LLMs can extract and classify evidence corresponding to predefined double-entry accounting relationships underlying the text.
ProofPaper ↗
Key points
- Financial disclosures express the underlying relationships of double-entry accounting through unstructured text.
- While financial LLM benchmarks predominantly emphasize question answering and classification, schema-guided evidence discovery remains relatively under-explored.
- Using a financial accounting graph comprising 11 nodes (accounts) and 11 edges (transaction types), we evaluate mid-sized models from 5 open-weight families across 1,300 SEC 10-K Item 7 filings spanning 26 years, comparing their extraction performance against three proprietary models.
- Building on this study, we introduce a human-annotated relation classification benchmark comprising 483 passages across three financial accounting categories, evaluating 34 open-weight and 10 proprietary models.
Sources (1)
- [1]FinGREAT: Evaluating LLMs on Financial Graph-guided Retrieval And TypingarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 09:57 PM
We investigate whether open-weight LLMs can extract and classify evidence corresponding to predefined double-entry accounting relationships underlying the text.
Financial disclosures express the underlying relationships of double-entry accounting through unstructured text.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 13, 2026Plan or Retrieve First? A Scale Study of Tool Discovery Architectures
- Oct 13, 2026Google Deepmind’s New Resident Philosopher Tackles AI Ethics
- Oct 13, 2026Training LLMs for Multi-Agent Collaboration with Verifiable Information Rewards
- Oct 13, 2026Copy-Paste Language Models
- Oct 13, 2026Beyond the Verdict: Agentic Code Judging with Verifiable Feedback
- Oct 13, 2026SHELDON: Reasoning through Dual Search over Hypothesis and Observation Spaces