Information-Theoretic Foundations of Online Temperature Sampling

Key points

• By Kac's lemma and the AEP, a stationary source takes time exponential in its entropy to repeat a block.1
• Our contribution is the online problem: the conditionals $p1, p2, \dots$ are revealed one at a time, and the sampling distribution $qt$ must be chosen causally from $p1, \dots, pt$ while the budget couples the entire horizon.1
• We propose Entropic Online Sampling (EOS), which re-solves for the temperature on each prefix.1
• We prove it loses $O(R\log n)$ entropy against the offline optimum while overshooting the budget by $O(R\log n)$, with $R$ an instance-dependent drift parameter.1

Sources (1)

• [1]Information-Theoretic Foundations of Online Temperature Sampling
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 10, 04:29 PM

  Repetitive, degenerate generation is a persistent failure of LLM decoding, and the standard remedies (temperature, nucleus, and top-$k$ sampling) are static heuristics.

  By Kac's lemma and the AEP, a stationary source takes time exponential in its entropy to repeat a block.

Extractive summary: sentences quoted from the sources.