LiveAct: Towards Real-Time Multi-Agent World Models with Heterogeneous Action Interfaces

Key points

• Interactive video world models predict how an environment evolves under user actions, but most existing systems assume one controlled agent and one task-specific action interface.1
• We introduce LiveAct, centered on a Heterogeneous Action Interface (HAI) whose two components mirror this decomposition.1
• We further transfer the complete interface to a blockwise causal generator through teacher forcing and rollout-aware distribution matching, enabling efficient streaming generation while retaining agent-specific control and action semantics.1
• We instantiate the formulation on six multi-agent sources spanning games and multi-robot manipulation, and evaluate visual quality, action following, agent attribution, and latency.1

Sources (1)

• [1]LiveAct: Towards Real-Time Multi-Agent World Models with Heterogeneous Action Interfaces
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 08:34 AM

  We study multi-agent video world modeling, where one model generates a joint visual future for multiple controlled agents while interpreting heterogeneous controls ranging from binary game buttons to continuous robot commands.

  Interactive video world models predict how an environment evolves under user actions, but most existing systems assume one controlled agent and one task-specific action interface.

Extractive summary: sentences quoted from the sources.