No Single Tokenizer Feature Reliably Predicts Downstream Language Model Performance

Key points

• Tokenization is a critical component of the language modeling pipeline, but it is poorly understood.1
• Tokenizers can impact downstream performance, but there is currently mixed evidence about which tokenizer features predict a better language model without requiring a full training run.1
• We evaluate both models and their tokenizers and test whether there is a relationship between any proposed intrinsic tokenizer evaluations and model performance, as measured by a wide variety of benchmarks.1
• We propose novel linguistically grounded metric introduced in this work shows that new metrics can capture aspects of tokenizer quality that existing metrics miss, though their predictive power is likely to vary across model families, languages, and tasks.1

Sources (1)

• [1]No Single Tokenizer Feature Reliably Predicts Downstream Language Model Performance
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 11:05 AM

  Despite the quantity and variety of proposed intrinsic tokenizer metrics, we find that none consistently predicts downstream performance across model families, tasks, and languages.

  Tokenization is a critical component of the language modeling pipeline, but it is poorly understood.

Extractive summary: sentences quoted from the sources.