Copying Under Repetition: From Prefix Matching to Position Hashing

Key points

• Frontier LLMs can copy random text almost flawlessly, yet they often overrun or truncate long blocks of repeated characters.1
• In repeated strings, matching local content no longer reliably identifies where copying should continue or when it should stop.1
• In contrast, we prove a width-independent impossibility result for copying under sequential finite-precision softmax accumulation.1
• If this threshold grows sublinearly with $N$, these models cannot both complete a long all-identical copy and stop correctly, a pattern resembling the termination errors of frontier LLMs. In synthetic copying experiments, baseline transformer models fail on long repeated strings even within the training range.1

Sources (1)

• [1]Copying Under Repetition: From Prefix Matching to Position Hashing
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 04:50 PM

  Nevertheless, we show that a two-layer transformer with $O(\log\log N)$-bit precision and $O(\log N/\log\log N)$ width can copy every string up to a chosen design horizon $N$ and terminate correctly under arbitrary repetition.

  Frontier LLMs can copy random text almost flawlessly, yet they often overrun or truncate long blocks of repeated characters.

Extractive summary: sentences quoted from the sources.