Failure Modes of Vision-Language Models for Human Activity Understanding in Thermal HRI

Key points

• As vision-language models (VLMs), originally developed for RGB imagery, are increasingly integrated into embodied systems for scene understanding and high-level planning, thermal sensing offers a promising way to extend activity perception to low-light and privacy-sensitive settings.1
• However, the failure modes of VLMs in thermal human activity understanding for human-robot interaction (HRI) remain underexplored.1
• We fill this gap by evaluating seven locally deployable VLMs, including models developed for physical AI and robotics, on two person-centric thermal datasets.1
• Our diagnostic tasks examine fine-grained action recognition, activity timeline reconstruction, and the ability to withhold unsupported answers.1

Sources (1)

• [1]Failure Modes of Vision-Language Models for Human Activity Understanding in Thermal HRI
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 05:49 AM

  Thermal cameras enable robots to perceive human activities in darkness while revealing less personally identifiable appearance information and fewer details of the surrounding environment than RGB cameras.

  As vision-language models (VLMs), originally developed for RGB imagery, are increasingly integrated into embodied systems for scene understanding and high-level planning, thermal sensing offers a promising way to extend activity perception to low-light and privacy-sensitive settings.

Extractive summary: sentences quoted from the sources.