Copy-Paste Language Models
We propose Copy-Paste Language Models (CPLMs), which add a lightweight Copy-Paste head to the conventional language modeling (LM) head.
ProofPaper ↗
Key points
- Transformers are intuitively considered to be efficient at copying from their inputs.
- We actually show that the ability of LLMs to copy in context can be improved through explicit modeling.
- We validate our approach by pretraining models from 135M to 1.7B parameters, showing improvements across all scales and backbones with negligible training and inference overheads.
- We explain the success of CPLMs by showing that they partially mitigate the LM head gradient bottleneck through an alternative gradient path.
Sources (2)
- [1]Copy-Paste Language ModelsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 10:03 PM
We propose Copy-Paste Language Models (CPLMs), which add a lightweight Copy-Paste head to the conventional language modeling (LM) head.
Transformers are intuitively considered to be efficient at copying from their inputs.
- [2]Copy-Paste Language ModelsHugging Face Daily Papers · Oct 13, 12:00 AM · same content
Extractive summary: sentences quoted from the sources.
Before this
- Oct 12, 2026MLAfication: Efficiently Enabling High-Fidelity MLA for Modern LLMs
- Oct 12, 2026Can we change an AI model’s architecture without retraining it from scratch?
- Oct 12, 2026Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged
- Oct 11, 2026E8 Lattice Key Quantization for Low-Bit KV-Cache Compression
- Oct 11, 2026History Replays Itself: Unleashing Long-Context Capability in Linear Attention
- Oct 10, 2026Retraining-Free Depth Pruning of Vision Transformers: A Cross Domain Study of Layer Redundancy