RealityBench: Simulated Environments to Evaluate a Vision Language Model's Ability to Document Human Physical Work

Key points

• One key supporting capability is to understand what the worker is doing, seeing through the human's camera or smart glasses, and assist them in documentation.1
• Here, we model real-world physical work in Blender across eleven environments, incorporating variants in both image acceptability and documentation.1
• The acceptability task asks whether a particular view would be suitable proof-of-work and documentation asks whether an agent can understand the task-relevant state of the environment from that view.1
• Across 26 vision-language models, we find that top models largely solve documentation when the answer is text in frame, but are weak at reading measurements and at judging small parts and spatial relations, such as whether a tape measure leans.1

Sources (1)

• [1]RealityBench: Simulated Environments to Evaluate a Vision Language Model's Ability to Document Human Physical Work
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 11:41 PM

  Augmenting worker capabilities in the physical world through guidance has been a decades-long goal for AI agents.

  One key supporting capability is to understand what the worker is doing, seeing through the human's camera or smart glasses, and assist them in documentation.

Extractive summary: sentences quoted from the sources.