Functional Cross-layer Experts Routing for One-step Diffusion Distillation

Key points

• One-step distillation compresses diffusion trajectories into a single forward evaluation, yet how internal capacity should be structured within that remaining pass remains largely unexplored.1
• We formalize this challenge as functional allocation.1
• Our analysis reveals that decoupling representations is beneficial only when expert interference outweighs shared transfer, favoring an always-active shared backbone augmented with conditionally routed residuals over standard isolated MoE architectures.1
• Across COCO-10K, DeMoE establishes state-of-the-art fidelity and diversity among single-step models, demonstrating that structured internal computation reallocation provides a superior compression paradigm for diffusion models.1

Sources (1)

• [1]Functional Cross-layer Experts Routing for One-step Diffusion Distillation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 03:46 AM

  We propose DeMoE, which pairs a persistent shared expert with conditionally routed expert slices across image FFNs, trained under a one-step distribution-matching objective with executed-route supervision.

  One-step distillation compresses diffusion trajectories into a single forward evaluation, yet how internal capacity should be structured within that remaining pass remains largely unexplored.

Extractive summary: sentences quoted from the sources.