APRiL: An Open-ended Benchmark for Chinese Social Awareness in Large Language Models

Key points

• Large language models are increasingly used in social interactions, where appropriate responses require recognizing implicit cues, inferring mental states, identifying conflicts, and considering potential consequences.1
• APRiL contains 2,196 Social Awareness Evaluation Instances across six categories and two complementary tasks: Social Scenario Analysis evaluates models' understanding of social cues, mental states, and conflict structures, while Socially Aware Response assesses whether they can translate this understanding into appropriate, natural responses.1
• Experiments on 12 models from six families reveal a consistent understanding-to-response gap.1
• APRiL provides a systematic resource for advancing socially aware language models in Chinese contexts.1

Sources (1)

• [1]APRiL: An Open-ended Benchmark for Chinese Social Awareness in Large Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 06:28 PM

  To address this gap, we introduce APRiL, an open-ended benchmark for Chinese social awareness with instance-specific rubrics.

  Large language models are increasingly used in social interactions, where appropriate responses require recognizing implicit cues, inferring mental states, identifying conflicts, and considering potential consequences.

Extractive summary: sentences quoted from the sources.