Truncate or Not? A Scaling Study of Looped Transformers with Truncated Backpropagation
Looped transformers reuse a shared block of layers to increase computational depth without increasing the number of trainable parameters.
ProofPaper ↗
Key points
- They are trained with backpropagation through time (BPTT).
- To save training compute, BPTT is often truncated to the last few loops, although this may weaken the benefit of looping.
- To study this tradeoff systematically, we jointly vary model size, training tokens, and forward and backward loop counts across nearly 600 training runs and fit a unified scaling law that captures the effects of looping and truncation through effective parameters.
- In matched-compute runs, full BPTT through fewer loops consistently outperforms truncated settings of the same compute.
Sources (1)
- [1]Truncate or Not? A Scaling Study of Looped Transformers with Truncated BackpropagationarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 08:52 AM
Looped transformers reuse a shared block of layers to increase computational depth without increasing the number of trainable parameters.
They are trained with backpropagation through time (BPTT).
Extractive summary: sentences quoted from the sources.
Before this
- Oct 19, 2026Internalizing Chain-of-thought into Looped Transformers
- Oct 19, 2026TorchNeuron: running HF models on AWS Trainium is easier than ever
- Oct 19, 2026SpecKCast: Spectral Kalman Forecasting for Diffusion Acceleration
- Oct 18, 2026SlimFormer++: Token Mixing Where It Matters for Efficient 3D Medical Image Segmentation
- Oct 17, 2026TopoCompile: Molecular Generation with Exact Global Count Constraints
- Oct 16, 2026Hugging Face ❤️ TorchTPU: TPUs, Natively