PerSo: Investigating Facets of LLMs' Social Intelligence in Real-World Persuasion Scenarios

Key points

• Social intelligence comprises two aspects: cognitive intelligence, the ability to infer others' mental states, and behavioral intelligence, the ability to select appropriate social actions based on these inferences to achieve social goals.1
• Persuasion provides a natural testbed in which both aspects are simultaneously at play, yet prior benchmarks typically evaluate them in isolation using synthetic scenarios.1
• We introduce PerSo, a benchmark that evaluates both aspects on real-world persuasion dialogues from Reddit's ChangeMyView community, using an LLM-in-the-loop pipeline to construct the evaluation tasks, validated through human annotation.1
• Our evaluation of diverse LLMs shows that performance generally scales with model size but varies across dimensions.1

Sources (1)

• [1]PerSo: Investigating Facets of LLMs' Social Intelligence in Real-World Persuasion Scenarios
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 05:39 PM

  Understanding the social intelligence of large language models (LLMs) is increasingly important for applications involving human-AI interaction and multi-agent environments.

  Social intelligence comprises two aspects: cognitive intelligence, the ability to infer others' mental states, and behavioral intelligence, the ability to select appropriate social actions based on these inferences to achieve social goals.

Extractive summary: sentences quoted from the sources.