ResearchResearch paperLarge Language Models · Training & Scaling · Efficiency & Inference1 source · Oct 20, 2026

How Much Gradient Does Truncated Backprop Lose in Looped Transformers?

Looped transformers reuse one block of layers many times and are often trained by backpropagating through only the last k loops.

We measure how much of the weight gradient this throws away, exactly and for every k from one backward pass, on nine looped language models, and report three findings.
First, on a trained model this lost gradient is set near the output: once the loop count is well above k, it barely changes with the loop count (within about 0.03 from 32 to 64 loops on the three models we tested, as a pre-registered test predicted for Parcae).
Where the backward signal shrinks, $ρ^k$, with $ρ$ the signal's fitted shrink per loop, approximates it at moderate loop counts but overstates it at long ones, because the fitted $ρ$ drifts up as the fit takes in more loops.
So a trained model's value overstates what truncation removes later.

[1]How Much Gradient Does Truncated Backprop Lose in Looped Transformers?
arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 07:42 PM
Looped transformers reuse one block of layers many times and are often trained by backpropagating through only the last k loops.
We measure how much of the weight gradient this throws away, exactly and for every k from one backward pass, on nine looped language models, and report three findings.

Extractive summary: sentences quoted from the sources.

Before this