PixelMoE: Scaling Pixel-Space Diffusion with LLM-Style Mixture-of-Experts

Key points

• In this work, we show that standard LLM-style Mixture-of-Experts (MoE) architectures can be effectively transferred to pixel-space diffusion.1
• Building on this finding, we introduce PixelMoE, a pixel-space generative framework that sparsely scales model capacity with standard LLM-style MoE, without diffusion-specific routing mechanisms or auxiliary routing objectives.1
• To complement sparse capacity scaling, PixelMoE further introduces Pixel-Space Hierarchical Flow Matching, which varies spatial resolution along the denoising trajectory from global structure formation to fine-grained detail refinement.1
• On class-conditional ImageNet-1K, PixelMoE achieves FID scores of 1.47 at 256x256 and 1.53 at 512x512 without representation alignment or perceptual losses, comparing favorably with leading pixel-space and latent-space generative models.1

Sources (1)

• [1]PixelMoE: Scaling Pixel-Space Diffusion with LLM-Style Mixture-of-Experts
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 05:59 PM

  Pixel-space diffusion models provide an appealing alternative to latent-space diffusion models by directly modeling raw images without relying on a learned tokenizer.

  In this work, we show that standard LLM-style Mixture-of-Experts (MoE) architectures can be effectively transferred to pixel-space diffusion.

Extractive summary: sentences quoted from the sources.