Fully Interpretable Minimal Transformers: From Geometry to Algorithm
We present a framework for building and interpreting minimal transformer models.
ProofPaper ↗
Key points
- By constraining a transformer's embedding dimension and head size to 2, we enable full two-dimensional visualization of its internal representations.
- We train a transformer on a simple task where it must produce the most recently observed even number whenever the '+' operator appears in a sequence of digits.
- We introduce a suite of interpretability visualizations that make the algorithmic interpretation of this procedure explicit.
- Our framework offers a pedagogical and experimental testbed to explore how transformers use informational geometry to implement next-token prediction.
Sources (1)
- [1]Fully Interpretable Minimal Transformers: From Geometry to AlgorithmarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 11:01 AM
We present a framework for building and interpreting minimal transformer models.
By constraining a transformer's embedding dimension and head size to 2, we enable full two-dimensional visualization of its internal representations.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 7, 2026Iris-3B: Going Beyond the Latent with Pixel-Space Diffusion Training, Conversion and Fine-Tuning
- Oct 6, 2026openbmb/MiniCPM-V-4.7-35B-A3B-gguf
- Oct 6, 2026EmbeddingGemma 2: an open, lightweight multimodal embedding model
- Oct 5, 2026LiquidAI/d1-omni-600M
- Oct 5, 2026MC-Sparse: Deconstructing and Closing the Dense-Sparse Attention Gap in Diffusion Transformers
- Sep 30, 2026Cloudflare/clef-flash