World-T3: Test-Time Training as Persistent Memory for Real-Time Video World Models

Key points

• Recent advances in video world models have enabled plausible interactive environments, yet long-horizon generation places a fundamental demand on visual memory: the previously observed world must remain consistent as the model continues to generate.1
• Existing autoregressive approaches often implicitly rely on the generative model itself to retain such information, leading to spatial memory decay beyond the active context.1
• To robustly update this memory, we propose a recursive least-squares rule that incrementally absorbs novel visual features via online pose-keyed memory updates.1
• Our TTT memory reads out previously observed scene features from the fast-weight state by camera pose, enabling consistent generation with constant storage and per-step computation regardless of rollout length.1

Sources (1)

• [1]World-T3: Test-Time Training as Persistent Memory for Real-Time Video World Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 09:52 PM

  We introduce World-T3, a camera-controlled video world model with a test-time training memory that consolidates past observations into dedicated model parameters.

  Recent advances in video world models have enabled plausible interactive environments, yet long-horizon generation places a fundamental demand on visual memory: the previously observed world must remain consistent as the model continues to generate.

Extractive summary: sentences quoted from the sources.