InstantFusion: A Shared Latent Interface for Heterogeneous Image Generators

Key points

• Diffusion/flow models have emerged as the de facto paradigms for image generation, giving rise to a diverse family of modern generators with increasingly heterogeneous architectures and representations.1
• Despite their remarkable generation capabilities, the cross-model alignment of latent representations and the consistency of their underlying denoising dynamics remain underexplored.1
• In this work, we uncover an intriguing cross-model compatibility: independently trained generators exhibit similar latent spatial organization, while directly handing off intermediate states between unaligned models can still preserve coarse visual content.1
• Specifically, to enable flexible cross-model state translation, model-specific Latent AutoEncoders (LAEs) are introduced to project intermediate states from different generators into a shared latent space and decode them into the latent space of a target generator.1

Sources (2)

• [1]InstantFusion: A Shared Latent Interface for Heterogeneous Image Generators
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 05:59 PM

  Motivated by these observations, we propose InstantFusion, a lightweight shared latent interface that connects the denoising trajectories of heterogeneous image generators.

  Diffusion/flow models have emerged as the de facto paradigms for image generation, giving rise to a diverse family of modern generators with increasingly heterogeneous architectures and representations.

• [2]InstantFusion: A Shared Latent Interface for Heterogeneous Image Generators
  Hugging Face Daily Papers · Oct 9, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.