CARES: A Controlled Synthetic Benchmark of Speaker Reactions to Sound

Key points

• One difficulty is deciding which elements of the audio should be kept, since a description cannot include them all.1
• We focus on audio events and define sound salience with a simple rule: a sound is salient when a speaker audibly reacts to it.1
• We then benchmark six audio-language models on three tasks: identifying the scene, tagging the sounds present, and classifying reactions.1
• We show that these models hear the sounds but miss how the speakers react to them.1

Sources (1)

• [1]CARES: A Controlled Synthetic Benchmark of Speaker Reactions to Sound
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 03:05 PM

  Automatic audio scene description turns a recording into a text account of a situation.

  One difficulty is deciding which elements of the audio should be kept, since a description cannot include them all.

Extractive summary: sentences quoted from the sources.