Patch where the text is new: a compression view of where byte-level models should spend compute

Key points

• Byte-level language models such as the Byte Latent Transformer (BLT) run their large global model once per patch and start a patch where a small model's next-byte entropy is high.1
• Outside math the advantage shrinks: novelty lowers bits per byte on prose and ties entropy on source code and on synthetic execution traces.1
• Computed results have ordinary surprisal for the patcher's model, so surprisal-seeking rules pass over them: entropy concentrates patch starts on high-surprisal bytes most strongly of all rules (boundary enrichment 1.61) and is the worst on computed results, while the best rules sit near chance enrichment.1
• We also report what did not work: typed uncertainty as a training rule, switching rules after training, and a context-aware novelty predictor.1

Sources (1)

• [1]Patch where the text is new: a compression view of where byte-level models should spend compute
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 03:47 PM

  We propose novelty patching, motivated by LZ77-style compression: start a patch where the upcoming bytes are unlikely to repeat the recent context.

  Byte-level language models such as the Byte Latent Transformer (BLT) run their large global model once per patch and start a patch where a small model's next-byte entropy is high.

Extractive summary: sentences quoted from the sources.