Benefit and Cost of Random-Token Replacement under Repeated Data: A Known-Entropy Decomposition and Evidence from Ten Real Corpora
On a controlled source whose entropy rate is known in closed form, we show that replacement shrinks predictions toward the corpus average.
ProofPaper ↗
Key points
- When unique pretraining data is scarce, models see the same tokens many times and are thought to memorize.
- Random-token replacement, an input-corruption augmentation, is one remedy reported to lower validation loss, but a lower loss alone does not reveal which source of error it reduces.
- We also introduce a decomposition that separates validation loss into approximation, estimation, and memorization terms.
- It harms only the census corpus, program-generated JSON that is the most redundant of the ten and much more redundant than any of 64 further prose and code corpora we measure.
Sources (1)
- [1]Benefit and Cost of Random-Token Replacement under Repeated Data: A Known-Entropy Decomposition and Evidence from Ten Real CorporaarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 03:57 AM
On a controlled source whose entropy rate is known in closed form, we show that replacement shrinks predictions toward the corpus average.
When unique pretraining data is scarce, models see the same tokens many times and are thought to memorize.
Extractive summary: sentences quoted from the sources.