Making Language the Low-Rate Code of Vision: Successive Refinement from Words to Pixels

Key points

• An ordered visual code lets a short prefix stand in for an image, and the training loss decides what that prefix contains.1
• Under pixel reconstruction the first bits go to the directions of highest variance, and we prove that a caption can then be independent of the entire first half of a code that is rate-optimal for pixels at every length.1
• Putting words first then costs nothing among the caption stages exactly when the captions form a summarization chain, and it costs the pixels at most the caption uncertainty left in a pixel-optimal reconstruction, which is large for short prefixes and small at full length.1
• We build RUNG, an ordered code whose prefixes are supervised by a summarization ladder of captions and whose full length reconstructs the image; generation extends the code past the prompt and captioning truncates it.1

Sources (1)

• [1]Making Language the Low-Rate Code of Vision: Successive Refinement from Words to Pixels
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 10:32 AM

  We cast captioning and text-to-image generation as successive refinement under a distortion measure that changes along the code, from caption log-loss in the early stages to pixel distortion in the last, so that each stage is compared with its own optimum.

  An ordered visual code lets a short prefix stand in for an image, and the training loss decides what that prefix contains.

Extractive summary: sentences quoted from the sources.