Scaling Multilingual Pretraining of Pixel Language Models

Key points

• In this paper, we train a family of MPIXEL models along three axes of variation: more languages, more scripts, and larger model capacity under fixed per-language compute.1
• In evaluations on five tasks covering 83 languages, we find that scaling acts locally: each axis improves the added languages and scripts, while increased capacity mitigates the curse of multilinguality from pretraining on more languages.1
• We also show that larger pixel models reserve more layers for visual processing, and that cross-lingual alignment weakens as more languages are added.1
• Overall, our findings characterize how multilingual scaling improves pixel language models.1

Sources (1)

• [1]Scaling Multilingual Pretraining of Pixel Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 12:57 PM

  Pixel language models process text as rendered images instead of a finite vocabulary, which gives them strong cross-lingual transfer abilities.

  In this paper, we train a family of MPIXEL models along three axes of variation: more languages, more scripts, and larger model capacity under fixed per-language compute.

Extractive summary: sentences quoted from the sources.