Personalization Without Stereotyping: Do Speech LLMs Know When to Turn a Deaf Ear?

Key points

• Recent speech LLMs can leverage rich paralinguistic cues, enabling more complex and personalized interactions.1
• We evaluate eight recent speech LLMs and find that, despite often recognizing demographic cues, most fail to translate them into appropriate personalization.1
• Explicit demographic labels improve personalization but also amplify stereotypical associations already present in the textual backbones.1
• These findings highlight the need to train speech LLMs not only to perceive demographic cues, but to learn when they should shape their responses.1

Sources (1)

• [1]Personalization Without Stereotyping: Do Speech LLMs Know When to Turn a Deaf Ear?
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 01:06 PM

  We introduce BASIS-Bench, a benchmark designed to disentangle these two aspects through three axes: (1) demographic robustness, measuring disparities in content-based tasks across speakers and demographic classification capabilities; (2) stereotypical bias, measuring whether perceived demographic cues trigger unprompted stereotypical behavior; and (3) personalization, measuring whether models can

  Recent speech LLMs can leverage rich paralinguistic cues, enabling more complex and personalized interactions.

Extractive summary: sentences quoted from the sources.