Long-horizon Video Generation with Slot Memory Addressed By Region Tokens

Key points

• Memory for long-horizon video generation must satisfy two competing requirements: it must remain bounded so generation can continue indefinitely, while still recovering what was observed at a place when the camera returns, regardless of how long ago it was seen.1
• Reconstruction-based memories provide spatial recall by caching and reprojecting explicit geometry, but require growing storage and depend on estimated 3D structure.1
• Recurrent memories keep a constant-size state, but retention is governed by temporal recurrence, causing old observations to fade with time.1
• SMART stores content in a constant-size slot recurrence whose writes are addressed by the observing camera and whose reads are queried by the rendering camera, without constructing an explicit 3D representation or maintaining a growing cache.1

Sources (1)

• [1]Long-horizon Video Generation with Slot Memory Addressed By Region Tokens
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 10:28 PM

  We introduce SMART, a slot memory addressed by region tokens, which combines bounded storage with spatially addressed recall.

  Memory for long-horizon video generation must satisfy two competing requirements: it must remain bounded so generation can continue indefinitely, while still recovering what was observed at a place when the camera returns, regardless of how long ago it was seen.

Extractive summary: sentences quoted from the sources.