ALIAS: Evaluating Performance and Memorization in LLM-Written TLA+ Specifications
Memorization inflates the benchmark scores of large language models (LLMs), since a model can reproduce answers it saw in training.
ProofPaper ↗
Key points
- Researchers test for it with consistent identifier renaming (CIR), which gives every identifier in a task a new name, and treat a lower score on the renamed task as evidence of recall.
- We test this method on the Temporal Logic of Actions Plus (TLA+) language, whose model checker acts as an oracle by confirming whether a renamed task preserves the exact behaviors of the original.
- Across 111 TLA+-Bench tasks, renaming lowers the correctness of Claude Opus 4.8 from 28% to 18% and of GPT-5 from 9% to 7%.
- In two model families fine-tuned on disjoint halves of the tasks, renaming removes about half of the training gain in likelihood and costs 4.9 percentage points (Qwen) and 1.3 percentage points (Gemma) on unseen tasks.
Sources (1)
- [1]ALIAS: Evaluating Performance and Memorization in LLM-Written TLA+ SpecificationsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 02:02 PM
Memorization inflates the benchmark scores of large language models (LLMs), since a model can reproduce answers it saw in training.
Researchers test for it with consistent identifier renaming (CIR), which gives every identifier in a task a new name, and treat a lower score on the renamed task as evidence of recall.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 21, 2026OpenAI Frontier Models Arrive on Google Cloud and Model Garden
- Oct 20, 2026When Do LLMs Help with Feature Selection? A Systematic Empirical Study
- Oct 20, 2026FairStress: A deeper evaluation of fairness in LLM decision-making
- Oct 19, 2026The Culture Within: Towards Mechanistic Understanding of Cultural Reasoning in LLMs
- Oct 19, 2026Introducing Clairic, a new System One class of extraction models
- Oct 19, 2026Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM
