Gamma: Grounded Multi-Agent Memory with Mechanical Audit for Memory-Intensive Manipulation

Key points

• Yet the same policy, finetuned once to follow a privileged oracle's grounded-subgoal text, nearly solves these tasks: the difficulty lies entirely in mapping memory to a grounded symbolic subgoal.1
• Existing work entrusts this stage to a single VLM that selects keyframes and reasons over raw frames, and falls far short of the oracle: key identities and dynamics are buried in noisy pixels, the reasoning spans hundreds of frames in an unbounded context, and hallucination corrupts the memory itself.1
• On a sixteen-task memory-manipulation suite GAMMA recovers a large fraction of the oracle ceiling with the policy held fixed, reaching the ceiling on several tasks, and transfers with unchanged contracts to a second benchmark.1

Sources (1)

• [1]Gamma: Grounded Multi-Agent Memory with Mechanical Audit for Memory-Intensive Manipulation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 11:00 PM

  Pretrained vision-language-action (VLA) policies fail on memory tasks, behaviour that depends on events observed long before the current frame, because their training data is overwhelmingly Markov and mapping episodic memory directly to action chunks is hard.

  Yet the same policy, finetuned once to follow a privileged oracle's grounded-subgoal text, nearly solves these tasks: the difficulty lies entirely in mapping memory to a grounded symbolic subgoal.

Extractive summary: sentences quoted from the sources.