Evaluating Social Reasoning in Embodied Vision-Language Models

Key points

• Robotics-ER outperforms GPT-4o-mini on error/competence detection but performs worse on detailed social-attribute judgments; adding a visual frame helps one task while hurting the other.1
• Because the comparison spans model families, these differences cannot be attributed to embodiment specialization alone.1
• This matters for multi-robot systems, where a shared foundation model could cause systematic social-perception errors to influence multiple coordination decisions.1
• This work presents an initial evaluation, with broader cross-context and video-based analyses left for future work.1

Sources (1)

• [1]Evaluating Social Reasoning in Embodied Vision-Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 02:14 AM

  Foundation models now serve as reasoning layers for robot teams, but it remains unclear whether embodiment-specialized vision-language models (VLMs) preserve the social-reasoning capabilities of general-purpose VLMs. We evaluate Gemini Robotics-ER on SHREC, a benchmark of real human-robot social interactions, and compare it with published general-purpose VLM baselines.

  Robotics-ER outperforms GPT-4o-mini on error/competence detection but performs worse on detailed social-attribute judgments; adding a visual frame helps one task while hurting the other.

Extractive summary: sentences quoted from the sources.