Mem4Act: Learning Effective Latent Memory for Long-Horizon Robotic Manipulation

Key points

• Long-horizon robotic manipulation fundamentally requires memory: agents must retain task-relevant information from past observations and interactions and use it to guide future actions.1
• Mem4Act addresses three coupled challenges: how to efficiently cover long histories, what information memory should retain, and how to ensure the policy actually relies on memory.1
• To this end, it organizes history via multi-granularity temporal aggregation, preserves complementary semantic and perceptual information through question-answering supervision and key-frame DINO feature reconstruction, and employs memory-first two-stage training to encourage greater use of memory by the action policy.1
• Across RoboMME, RMBench, and four real-world manipulation tasks, Mem4Act achieves state-of-the-art performance on memory-dependent control, with success rates of 49.88%, 61.0%, and 81.5%, respectively.1

Sources (1)

• [1]Mem4Act: Learning Effective Latent Memory for Long-Horizon Robotic Manipulation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 05:51 PM

  We introduce Mem4Act, a framework for learning effective latent memory as a fixed-size representation explicitly optimized for long-horizon robotic control.

  Long-horizon robotic manipulation fundamentally requires memory: agents must retain task-relevant information from past observations and interactions and use it to guide future actions.

Extractive summary: sentences quoted from the sources.