Asymptotic Behavior of Transformers
In this paper we study the behavior of transformer networks generating text over long periods of time.
ProofPaper ↗
Key points
- We show that even a three-token, single-layer attention-only transformer can fail to converge.
- We then study a rich class of transformers with arbitrary finite depth, layer normalization, MLPs, RoPE encodings, and an arbitrary mix of local and global layers.
- Every transformer in this class has a softmax temperature threshold above which its empirical cache distribution converges to a unique limit, independent of initial conditions.
- We show further that, while each transformer has a temperature threshold, the supremum of this set is infinite, so there is no global minimal temperature that forces all transformers to converge.
Sources (1)
- [1]Asymptotic Behavior of TransformersarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 12:59 AM
In this paper we study the behavior of transformer networks generating text over long periods of time.
We show that even a three-token, single-layer attention-only transformer can fail to converge.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 14, 2026DAGFormer: Global Topology Prediction for Transformer Cross-Layer Connections
- Oct 14, 2026Player-1: Learning to Play from Synthetic Experience
- Oct 13, 2026Bytes to Threats: Byte-Sequence Transformer For Malware Detection
- Oct 13, 2026Run fast, private agents on-device with LFM2.5-24B-A2B
- Oct 13, 2026Gemini 4 Argon: Built to solve your hardest problems
- Oct 13, 2026black-forest-labs/flux-3-image-klein-comfy