Matrix-Game 4.0: Harnessing Video Models for Real-Time Playable Game Worlds

Key points

• Moving from controllable videos to genuinely playable worlds requires persistent world state, executable game rules, and low-latency interaction.1
• A lightweight game engine maintains world state and executes interaction rules, a multimodal coding agent translates player ideas into game code and dynamic scene captions, and a renderer based on autoregressive video model transforms engine-rendered low-poly proxies into detailed visuals, conditioned on dynamic captions and memory references.1
• Together with targeted inference optimizations, Matrix-Game 4.0 enables 480p, 60 FPS gameplay on a single NVIDIA RTX5090 with a key-press-to-display latency of 112ms.1
• We demonstrate customizable shooter and driving games with engine-enforced mechanics, and release code, model weights, and low-poly game assets to support research and creation of playable game worlds.1

Sources (2)

• [1]Matrix-Game 4.0: Harnessing Video Models for Real-Time Playable Game Worlds
  Hugging Face Daily Papers · Oct 20, 12:00 AM

  We outline a roadmap for the next generation of Matrix-Game that separates game execution from visual generation through an executable game harness, and introduce Matrix-Game 4.0 as the first system built on this design.

  Moving from controllable videos to genuinely playable worlds requires persistent world state, executable game rules, and low-latency interaction.

• [2]Matrix-Game 4.0: Harnessing Video Models for Real-Time Playable Game Worlds
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 10:41 AM · same content

Extractive summary: sentences quoted from the sources.