Dynamic ordering in sequence generation in Transformers: experiments on low-resource Tagging scenarios

Key points

• TagInsert lifts the L2R constraint by removing the decoder-side attention mask and enabling the model to decide a preferred order of tag generation at inference time.1
• We experiment on the common Part-of-Speech tagging and the more complex Combinatory Categorial Grammar (CCG) supertagging tasks on multiple languages and low-resource data settings.1
• We show that, although encoder-only models like BERT remain strong performers for tagging tasks, TagInsert frequently outperforms a L2R encoder-decoder model, achieves results comparable to encoder-only models under data-scarce conditions, and shows better results on the long-tail of rare categories.1
• Our analyses show that TagInsert alleviates error propagation typical of autoregressive models by learning to defer difficult predictions to later stages in the decoding process.1

Sources (1)

• [1]Dynamic ordering in sequence generation in Transformers: experiments on low-resource Tagging scenarios
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 01:36 PM

  This paper introduces TagInsert, a Transformer-based model that challenges the common left-to-right (L2R) generation paradigm in Transformers, for sequence labelling tasks where dependencies may not naturally align with a L2R order.

  TagInsert lifts the L2R constraint by removing the decoder-side attention mask and enabling the model to decide a preferred order of tag generation at inference time.

Extractive summary: sentences quoted from the sources.