SocialCoAct: Enabling and Benchmarking Egocentric Social Co-Action for VLM-Based Humanoid Robots

Key points

• Enabling humanoid robots to participate effectively in social interactions is important for their integration into human-centered environments.1
• We formulate egocentric social co-action as an end-to-end agent task and develop SocialCoAct-Agent, which transforms egocentric observations and dialogue history into verbal responses and atomic action sequences, aligns each action with its corresponding utterance span, and executes the resulting behaviors in MuJoCo and on a physical humanoid robot.1
• To evaluate the VLM capabilities required by this task, we construct SocialCoAct-Bench from continuous egocentric recordings of natural multi-party interactions.1
• SocialCoAct provides a foundation for developing and evaluating VLM-based humanoid agents that coordinate context-aware speech and action in complex social interactions.1

Sources (1)

• [1]SocialCoAct: Enabling and Benchmarking Egocentric Social Co-Action for VLM-Based Humanoid Robots
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 17, 08:33 AM

  We introduce SocialCoAct, a unified agent-centered framework for VLM-based egocentric social co-action.

  Enabling humanoid robots to participate effectively in social interactions is important for their integration into human-centered environments.

Extractive summary: sentences quoted from the sources.