Anisotropic Representation Dynamics in Transformers
We develop a Mixture-Gated Theory of Anisotropic Representation Dynamics that separates self-attention into value transport, query gating, and key gating.
ProofPaper ↗
Key points
- Small differences between internal representations can shrink or grow as they pass through a Transformer layer, affecting how information is preserved and how sensitive the model is to perturbations.
- Our analysis reveals how the geometry of attended representations constrains gating and how interactions between these mechanisms can reverse contraction induced by value transport.
- Experiments across four Transformer architectures validate these predictions through analytical and finite-forward comparisons, revealing distinct and joint query--key gating mechanisms and nontrivial switching subspaces.
- Our results provide a mechanistic and spectral understanding of how attention routing can alter local representation dynamics.
Sources (1)
- [1]Anisotropic Representation Dynamics in TransformersarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 10:09 AM
We develop a Mixture-Gated Theory of Anisotropic Representation Dynamics that separates self-attention into value transport, query gating, and key gating.
Small differences between internal representations can shrink or grow as they pass through a Transformer layer, affecting how information is preserved and how sensitive the model is to perturbations.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 16, 2026Welcome Beam: a 501B open model you can own, customize, and scale
- Oct 15, 2026Granite Switch 4.1: Chat, Decisions, and Speech from One Endpoint
- Oct 15, 2026RoleQ: Role-Aligned Scales for Converting Language Models to Ternary
- Oct 15, 2026tencent/Hy-Memory-30B-A3B
- Oct 15, 2026Local and Distributed Large LLM Inference with MLX and RDMA on Mac Studio
- Oct 14, 2026Mapping the Post-Transformer Architecture Landscape in 2026 [D]