JUICE: Unified Multimodal Modeling in Pretrained Representation Spaces

Key points

• Most unified multimodal models treat images and text asymmetrically: a pretrained language model generates text, an attached diffusion head generates images, and outputs that contain both are produced in a fixed order, so the first output cannot adapt to the second.1
• Pretrained encoder spaces hold both modalities in one continuous form and carry knowledge learned at scale, and recent work finds them effective as generative targets for vision and language.1
• Which modalities are clean and which are generated then decides the task: image generation, captioning, visual question answering, or joint generation.1
• Training on joint and conditional denoising together beats either alone, and semantic encoder spaces beat VAE latents and raw pixels as targets.1

Sources (1)

• [1]JUICE: Unified Multimodal Modeling in Pretrained Representation Spaces
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 06:25 PM

  We introduce JUICE (Joint Unified multimodal modeling In Continuous Encoder spaces), a single flow matching Transformer over frozen image and text representations, in which each modality has its own noise level.

  Most unified multimodal models treat images and text asymmetrically: a pretrained language model generates text, an attached diffusion head generates images, and outputs that contain both are produced in a fixed order, so the first output cannot adapt to the second.

Extractive summary: sentences quoted from the sources.