Beyond Model Bias: Cross-Entropy Based Rankings in Literary and Academic Texts Remain Stable across Small Language Models

Key points

• This study presents empirical evidence of the invariance of ranking based on cross-entropy related measures (e.g. mean, variance of each data-point), across four distinct small scale transformer-based models, for a given dataset of text sources.1
• We consider two stylistic environments: a corpus of contemporary poetry and a corpus of academic research papers.1
• We demonstrate that mean and variance of cross-entropy maintain robust cross-model rank correlation ($ρ> 0.90$) across both regimes.1
• Geometric analysis of the structurally constrained academic regime reveals that higher-order moments are sensitive to specific models, although strong agreement persists.1

Sources (1)

• [1]Beyond Model Bias: Cross-Entropy Based Rankings in Literary and Academic Texts Remain Stable across Small Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 04:13 PM

  While language models provide powerful mechanisms for evaluating textual predictability, it remains unclear whether the relative extents of LM surprisal remain stable across models.

  This study presents empirical evidence of the invariance of ranking based on cross-entropy related measures (e.g. mean, variance of each data-point), across four distinct small scale transformer-based models, for a given dataset of text sources.

Extractive summary: sentences quoted from the sources.