Scaling Unified Multimodal Models to Long-Horizon Interleaved Generation

Key points

• We develop a large UMM natively trained on interleaved multimodal data, with strong capabilities in image synthesis, text rendering, complex composition, and multimodal planning, and study how to extend these native capabilities to substantially longer autoregressive trajectories.1
• We introduce a visual-memory framework that maintains persistent visual references, selectively retrieves scene-relevant historical states while preserving the complete semantic history, and uses few-step self-rollout to adapt the model to imperfect generated histories.1
• We further introduce LongMIG-Bench for evaluating long-horizon text-to-multi-image and image-to-multi-image generation.1
• Our model achieves high-fidelity long-horizon interleaved generation with strong character and scene consistency, demonstrating how visual memory can scale native interleaved generation to substantially longer trajectories.1

Sources (1)

• [1]Scaling Unified Multimodal Models to Long-Horizon Interleaved Generation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 04:46 PM

  Unified multimodal models (UMMs) provide an interface for interleaved text-image generation, where generated visual states can directly inform subsequent reasoning and synthesis.

  We develop a large UMM natively trained on interleaved multimodal data, with strong capabilities in image synthesis, text rendering, complex composition, and multimodal planning, and study how to extend these native capabilities to substantially longer autoregressive trajectories.

Extractive summary: sentences quoted from the sources.