Efficiency Measured in English: Tokeniser Coverage Bounds Small Language Models on Indic Scripts
Small Language Models with quantisation and parameter-efficient adaptation are widely offered as the route to multilingual AI under compute constraints, and their efficiency is almost always reported as a single memory figure.
ProofPaper ↗
Key points
- We show that this figure is measured in English and does not transfer.
- Across a 131k-sentence corpus spanning eleven Indic scripts, TinyLlama-1.1B's tokeniser emits 1.34 subword tokens per Latin-script word and 22.60 per Kannada word.
- Because SentencePiece byte fallback never raises an unknown-token marker, this failure is entirely silent: TinyLlama's UNK rate is 0.0% on every script, including the one it reads as bytes.
- MuRIL, an Indic-vocabulary encoder with a fifth of TinyLlama's parameters, reads every script at 1.23-2.26 tokens/word with zero UNK and zero byte fallback.
Sources (1)
- [1]Efficiency Measured in English: Tokeniser Coverage Bounds Small Language Models on Indic ScriptsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 17, 06:59 PM
Small Language Models with quantisation and parameter-efficient adaptation are widely offered as the route to multilingual AI under compute constraints, and their efficiency is almost always reported as a single memory figure.
We show that this figure is measured in English and does not transfer.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 17, 2026deepseek-ai/DeepSeek-V4.1-Flash-1017
- Oct 17, 2026Nemotron 3.5 Bolt, Muse Spark Open, and a Fast LFM2.5 24B
- Oct 16, 2026Liquid AI Releases d1-ultra: A Multimodal Decision Model That Scores 65 on the AA Decision Index
- Oct 16, 2026Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline
- Oct 16, 2026Welcome Beam: a 501B open model you can own, customize, and scale
- Oct 15, 2026RoleQ: Role-Aligned Scales for Converting Language Models to Ternary