SpireBench: Evaluating Long-Horizon Strategic Play of Agentic Models with Slay the Spire
Agentic large language models now tackle complex tasks, yet strategic reasoning over long horizons is under-studied: most agentic benchmarks are short and episodic, and performance on long tasks is hard to dissect into stored knowledge, reasoning and luck.
ProofPaper ↗
Key points
- We introduce SpireBench, in which models play the complete commercial deck-builder Slay the Spire (StS), where early decisions decide long-term success.
- Across 20 large language models (LLMs) at the game's highest difficulty, frontier models beat heuristic baselines and match the average human player but remain far behind experts.
- What separates strong models from weak is how well they play each fight; all of them misjudge risk over a run, preserving health rather than spending it to grow stronger.
- A comprehensive, self-consistent renaming of every game term shows that part of their performance rests on prior knowledge of the game, and models do not improve from the memories they write across runs, which are often wrong and overly cautious.
Sources (1)
- [1]SpireBench: Evaluating Long-Horizon Strategic Play of Agentic Models with Slay the SpirearXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 06:18 PM
Agentic large language models now tackle complex tasks, yet strategic reasoning over long horizons is under-studied: most agentic benchmarks are short and episodic, and performance on long tasks is hard to dissect into stored knowledge, reasoning and luck.
We introduce SpireBench, in which models play the complete commercial deck-builder Slay the Spire (StS), where early decisions decide long-term success.
Extractive summary: sentences quoted from the sources.