MixLID: Token-Level Language Identification in Mixed-Language Text

Key points

• Processing multilingual text requires both identifying a broad range of languages and distinguishing the language of each token in inputs containing multiple languages.1
• MixLID follows a two-stage training process: it first learns to distinguish individual languages using the language labels of monolingual examples, then continues training on mixed-language data.1
• For monolingual identification, MixLID outperforms all baselines in both accuracy and macro F1 on four benchmarks within the language coverage shared with each baseline.1
• MixLID achieves high performance across a broad range of languages in both monolingual and mixed-language settings.1

Sources (2)

• [1]MixLID: Token-Level Language Identification in Mixed-Language Text
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 11:21 AM

  We introduce MixLID, a token-level language identification model supporting 111 language-script combinations.

  Processing multilingual text requires both identifying a broad range of languages and distinguishing the language of each token in inputs containing multiple languages.

• [2]MixLID: Token-Level Language Identification in Mixed-Language Text
  Hugging Face Daily Papers · Oct 12, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.