Does Morphological Correctness Matter for Tokenization? A Scrambled Control Study in Tagalog

Key points

• Studies routinely establish that it helps while assuming the boundaries must be linguistically correct.1
• Tagalog marks voice with infixes (s-um-ulat 'wrote', s-in-ulat 'was written'), so the root is not contiguous and no sequence of BPE merges can recover it.1
• We build MorphAustro-Tok, a tokenizer that emits the root as a contiguous unit followed by a positional infix token, and compare it against a corpus-matched BPE baseline and, at two of three fertility levels, against a scrambled-morpheme control that preserves morph count, invertibility and per-type consistency while randomising the pseudo-infix.1
• Across 33 runs, MorphAustro-Tok improves Tagalog machine translation over matched BPE by 0.84, 1.49 and 3.40 mean sentence-level chrF++ points.1

Sources (1)

• [1]Does Morphological Correctness Matter for Tokenization? A Scrambled Control Study in Tagalog
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 07:49 PM

  Typology-aware tokenizers are motivated by the claim that segmentation which respects morpheme boundaries preserves information that frequency-driven segmentation destroys.

  Studies routinely establish that it helps while assuming the boundaries must be linguistically correct.

Extractive summary: sentences quoted from the sources.