Embeddings Are Overpriced? A Frozen 16 MB Vocabulary for Small and Ternary Language Models

Key points

• In a small language model the vocabulary is the single largest part.1
• We replace it with one frozen table of 512-bit codes, 16 MB, with no trainable parameters, derived once from a pretrained embedding by principal-component projection and sign binarisation, and we train the same Gemma-shaped body from scratch on the same 2.05B tokens with and without it.1
• With a ternary BitNet b1.58 body the whole model is a 46 MB file, still ties the float model on LAMBADA and PIQA, and on a laptop CPU generates 615 to 740 tokens per second in 62 MB of memory, five to six times the speed of the float model's 8-bit GGUF in llama.cpp at a seventh of the memory.1
• The frozen table wins by 1.7 to 2.3 nats on rare words, accepts 10,000 new words after training with a 3% cost where the learned table collapses by 43%, and all of it is released with the weights, the data and every evaluation answer.1

Sources (1)

• [1]Embeddings Are Overpriced? A Frozen 16 MB Vocabulary for Small and Ternary Language Models
  Hugging Face Daily Papers · Oct 15, 12:00 AM

  In Gemma 3 270M the embedding table holds 168M of 268M parameters, and because quantisation methods leave the embedding in 16-bit, it is also most of the file once the rest of the model is compressed.

  In a small language model the vocabulary is the single largest part.

Extractive summary: sentences quoted from the sources.