MixTok: Adaptive Visual Tokenization with Global 1D Tokens and Sparse 2D Refinement

Key points

• Efficient visual tokenization requires balancing representation compactness with the preservation of fine-grained image content.1
• Conventional grid-based tokenizers allocate tokens uniformly across spatial locations despite non-uniform visual complexity, whereas compact 1D tokenizers aggregate global information but can struggle to preserve local details.1
• At the highest-scoring locations, 2D token features replace the spatial features recovered from the 1D representation, and a shared decoder reconstructs the image from the resulting mixed features.1
• On ImageNet 256 x 256, MixTok achieves highly competitive reconstruction and generation performance, with an rFID of 1.195 and a gFID of 1.95 at respective average budgets of approximately 128 and 131 content tokens per image.1

Sources (2)

• [1]MixTok: Adaptive Visual Tokenization with Global 1D Tokens and Sparse 2D Refinement
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 08:07 AM

  We introduce MixTok, a hybrid visual tokenizer that combines a compact 1D global representation with sparse 2D tokens for local refinement.

  Efficient visual tokenization requires balancing representation compactness with the preservation of fine-grained image content.

• [2]MixTok: Adaptive Visual Tokenization with Global 1D Tokens and Sparse 2D Refinement
  Hugging Face Daily Papers · Oct 9, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.