In-Place Indic Tokenizer Expansion and Continual Pretraining of Nemotron-3-Nano-30B-A3B

Key points

• We expanded the vocabulary in place from 131,072 to 163,072 entries using continued byte-pair encoding and AdaptMix rebalancing, reducing tokens per word by 38.6%.1
• We initialized new input and output rows by averaging their original constituent rows, then trained them on 90% Indic and 10% English text before updating the backbone.1
• Continual pretraining then progressed through broad training, high-quality annealing, and long-context training over approximately 1.27 trillion tokens.1
• Direct token addition underperformed continued byte-pair encoding.1

Sources (1)

• [1]In-Place Indic Tokenizer Expansion and Continual Pretraining of Nemotron-3-Nano-30B-A3B
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 11, 05:53 AM

  We adapted Nemotron-3-Nano-30B-A3B for eight Indian languages without changing its backbone architecture.

  We expanded the vocabulary in place from 131,072 to 163,072 entries using continued byte-pair encoding and AdaptMix rebalancing, reducing tokens per word by 38.6%.

Extractive summary: sentences quoted from the sources.