UniState: Unifying Memory and Intervention for Controllable World Simulation

Key points

• World models offer increasingly rich environments for exploration, yet intervening in a world while observing the consequences from freely chosen viewpoints remains challenging.1
• This representation unifies spatial memory and user interventions.1
• Users specify object motion directly within spatial memory, while independently choosing the camera trajectory from which the resulting world state is observed.1
• We develop training and sampling strategies that help generated videos follow the specified object motion and camera trajectory while preserving this capacity to generate responses beyond the prescribed transformations.1

Sources (1)

• [1]UniState: Unifying Memory and Intervention for Controllable World Simulation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 18, 05:12 PM

  We tackle these challenges by introducing UniState, a framework for controllable world simulation built on a unified world-state representation.

  World models offer increasingly rich environments for exploration, yet intervening in a world while observing the consequences from freely chosen viewpoints remains challenging.

Extractive summary: sentences quoted from the sources.