Estimating Surprisal from the Speech Signal

Key points

• Although language is predominantly spoken, empirical tests of surprisal theory rely almost entirely on written text.1
• This raises a key question: should surprisal be defined over realization-specific acoustic inputs or over abstract linguistic units?1
• Obtaining surprisal from several speech LMs, we consistently find that surprisal estimated via importance sampling better predicts pupil dilation than realization-specific surprisal for words.1
• While we find that surprisal can be estimated without recourse to text, and that marginalizing over distinct realizations of words is more predictive of lexical processing effort, surprisal as a whole is not a strong predictor of change in pupil size, regardless of its source.1

Sources (1)

• [1]Estimating Surprisal from the Speech Signal
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 02:14 PM

  To address this, we introduce an importance sampling estimator that approximates the probability of an abstract unit by marginalizing over realizations generated by a text-to-speech model.

  Although language is predominantly spoken, empirical tests of surprisal theory rely almost entirely on written text.

Extractive summary: sentences quoted from the sources.