Fear Is Not Always Fear: Variation in Emotion Label Equivalence Across Datasets

Key points

• We test this assumption for five labels shared by GoEmotions, XED, and BRIGHTER, comparing models on identical inputs and under cross-corpus transfer.1
• What recalibration cannot repair is larger: transferred models lose 55% of native discrimination in aggregate, agreement in ranking between classifiers is only weakly related to that loss, and for surprise the resources disagree about which texts express the emotion.1
• The choice of training resource can therefore change conclusions: on two external tweet collections, BRIGHTER- and XED-trained classifiers give opposite answers about which shows more surprise, even after rank normalization, with intervals excluding zero in both directions.1
• A shared emotion label name does not establish an interchangeable measurement target.1

Sources (1)

• [1]Fear Is Not Always Fear: Variation in Emotion Label Equivalence Across Datasets
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 12:55 AM

  Emotion classifiers are routinely applied beyond their training data, on the assumption that same-named labels such as fear or surprise mean the same thing across resources.

  We test this assumption for five labels shared by GoEmotions, XED, and BRIGHTER, comparing models on identical inputs and under cross-corpus transfer.

Extractive summary: sentences quoted from the sources.