DialogMem: Evaluating Dialogue-Grounded Embodied Memory in 3D Environments

Key points

• Embodied agents in human-centered environments require memory that integrates what they observe with what humans tell them over time.1
• However, existing embodied memories are built primarily from visual experience, leaving open how dialogue with humans can be grounded in the objects encountered throughout that experience.1
• In experiments, appending dialogue to visual memory at evaluation time substantially improves QA but yields negligible navigation gains, suggesting that relevant dialogue is present in memory but not grounded to specific objects.1
• These results identify three bottlenecks for dialogue-grounded embodied memory: grounding and organizing dialogue at object-level entries during formation, retrieving the correct entry, and localizing it accurately.1

Sources (1)

• [1]DialogMem: Evaluating Dialogue-Grounded Embodied Memory in 3D Environments
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 17, 04:19 PM

  We introduce DialogMem, a two-phase benchmark in which agents build memory from time-aligned streams of posed RGB-D observations and dialogue, and later use it for instance-level object navigation and question answering (QA).

  Embodied agents in human-centered environments require memory that integrates what they observe with what humans tell them over time.

Extractive summary: sentences quoted from the sources.