Rethinking Video Autoencoders for Interactive World Models

Key points

• We build CADRE, an open family of causal diffusion representation autoencoders spanning multiple spatio-temporal compressions.1
• We train the same world model on each autoencoder in two domains, manipulation and navigation, with two environments each.1
• In robot manipulation, temporal compression is more effective in the autoencoder than in the world model.1
• Manipulation and navigation call for opposite compressions: temporal compression costs little when the camera is still, and spatial compression when it moves.1

Sources (1)

• [1]Rethinking Video Autoencoders for Interactive World Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 05:58 PM

  Video world models predict future frames from past frames and actions.

  We build CADRE, an open family of causal diffusion representation autoencoders spanning multiple spatio-temporal compressions.

Extractive summary: sentences quoted from the sources.