Modaliff: A Unified Token Diffusion Model for Any-to-Any Generation

Key points

• Any-to-any generation seeks a single model that translates flexibly among text, image, and audio, yet existing approaches either require fully aligned multimodal tuples or a shared representation that suppresses modality-specific structure.1
• Modaliff assembles conditions and targets into a unified token sequence, with per-token attributes to disambiguate token identities and role-based positional embedding to preserve each modality's internal structure across sequence layouts.1
• This design enables a single denoiser to support diverse generation tasks without task-specific architectural changes or a shared cross-modal latent space.1
• Experiments across text, image, and audio show that Modaliff achieves the best fidelity among any-to-any models on most generation tasks while using substantially fewer parameters and training data.1

Sources (1)

• [1]Modaliff: A Unified Token Diffusion Model for Any-to-Any Generation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 18, 09:07 PM

  We introduce Modaliff, a novel token-based diffusion model that learns from widely available bimodal pairs while preserving each modality in its native latent sequence.

  Any-to-any generation seeks a single model that translates flexibly among text, image, and audio, yet existing approaches either require fully aligned multimodal tuples or a shared representation that suppresses modality-specific structure.

Extractive summary: sentences quoted from the sources.