Multimodal Co-Generation With Continuous Bitstream Diffusion

Key points

• Across image-text, speech-text and protein sequence-structure modeling, models are trained separately from scratch, but each uses a single input projection, transformer backbone, bitwise prediction head and binary score-matching objective shared across its modalities.1
• We first validate the approach on MNIST-Sum, a controlled image-text task pairing raw binary images with equations encoded as text, achieving near-perfect conditional accuracy and high joint consistency without learned tokenizers.1
• In protein modeling, a single CoBit checkpoint supports forward folding, inverse folding, and joint sequence-structure generation.1
• Together, these results show that CoBit provides a common generative framework for substantially different modalities and forms of cross-modal dependence, without introducing a domain-specific objective or sampling procedure.1

Sources (1)

• [1]Multimodal Co-Generation With Continuous Bitstream Diffusion
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 09:54 PM

  We show that continuous bitstream diffusion (CoBit), originally developed for language modeling, provides a unified framework for joint and conditional multimodal generation using the same training and sampling protocol.

  Across image-text, speech-text and protein sequence-structure modeling, models are trained separately from scratch, but each uses a single input projection, transformer backbone, bitwise prediction head and binary score-matching objective shared across its modalities.

Extractive summary: sentences quoted from the sources.