ViMoLa: Discrete Motion Tokens for Language-Guided Video Generation

Key points

• Language-guided video generation often captures scene content while missing the requested motion.1
• A spatial vector-quantized autoencoder compresses dense trajectories into 64 tokens from a 512-entry codebook; a 28M-parameter transformer predicts the tokens from text; and zero-gated cross-attention adapters condition every frozen Wan 2.2 DiT block through forward hooks.1
• Trained on 462K videos from Something-Something v2 and Kinetics-400, ViMoLa raises directional accuracy from 0.287 to 0.442 (chance 0.250), with similar measured generative quality.1
• A low-rank adapter reaches 0.428 at 1.45% additional forward-pass FLOPs, and reusing the tokenizer and language model with newly trained CogVideoX adapters raises accuracy from 0.301 to 0.398.1

Sources (1)

• [1]ViMoLa: Discrete Motion Tokens for Language-Guided Video Generation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 18, 11:54 PM

  We present ViMoLa (Video Motion Language), which predicts a motion prior from language and injects it into a frozen image-to-video backbone.

  Language-guided video generation often captures scene content while missing the requested motion.

Extractive summary: sentences quoted from the sources.