ToMStories: Interpreting Theory-of-Mind in Synthetic Narratives

Key points

• Understanding people often requires inferring what they feel or want from how they behave rather than from what they state outright.1
• We ask whether language models can make these inferences, and how they do so.1
• Evaluating thirteen open-weight models (1B-72B parameters), we find that the largest models match humans on multiple choice but not open-ended questions.1
• Our analyses point to simplifications models make in both interpretation and generation of implicit psychological states, indicating that synthetic data can help us understand how models perform interpretation, but likely overestimates how well models interpret humans.1

Sources (1)

• [1]ToMStories: Interpreting Theory-of-Mind in Synthetic Narratives
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 08:38 PM

  We introduce Theory-of-Mind Stories (ToMStories), an evaluation dataset of 6,000 short synthetically generated narrative vignettes, each paired with a multiple-choice or open-ended question about a character's emotion, fear, motivation, or emotional wound.

  Understanding people often requires inferring what they feel or want from how they behave rather than from what they state outright.

Extractive summary: sentences quoted from the sources.