Rarity Determines Minimax Rates of Semantic Entropy From Weak Pairwise Judgments With a Known Channel

Key points

• Given the meanings of $n$ answers, a known channel declares each pair equivalent independently, with probability $q+Δ$ within a meaning and $q$ across meanings.1
• For a weak judge, with $nΔ^2\to0$, the global minimax rate is set by distributions near a vertex of the simplex and overstates the difficulty elsewhere.1
• We show that the difficulty is set by how many meanings can be rare.1
• The case $s=K-1$ gives the global rate $N^{-1/(K-1)}$ and the consistency threshold: with $Δ=n^{-γ}$ the entropy is uniformly consistently estimable exactly when $γ<K/(2K-2)$.1

Sources (1)

• [1]Rarity Determines Minimax Rates of Semantic Entropy From Weak Pairwise Judgments With a Known Channel
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 08:44 AM

  The semantic entropy of a language model is the entropy of the meanings of its sampled answers.

  Given the meanings of $n$ answers, a known channel declares each pair equivalent independently, with probability $q+Δ$ within a meaning and $q$ across meanings.

Extractive summary: sentences quoted from the sources.