Active Learning for Finding a Needle in a Haystack

Key points

• Training a classifier to find a rare class of documents, such as hate speech directed at a particular religious group, requires labelled examples that are expensive to obtain when the class makes up only a small fraction of a corpus.1
• Active learning reduces this cost, but it must start from an initial labelled set that contains examples of the target class.1
• We evaluate whether active learning can correct this bias.1
• On Measuring Hate Speech, uncertainty-based strategies recover less of the omitted subgroups than random sampling, whereas discriminative active learning recovers 55% of them without a significant loss of F1.1

Sources (1)

• [1]Active Learning for Finding a Needle in a Haystack
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 10, 08:30 PM

  On AG News and Measuring Hate Speech, at a prevalence of around 1%, we compare random sampling with eight active learning strategies, starting from initial sets that cover or omit known subgroups of the target class, and measure recall on the omitted subgroups as well as F1.

  Training a classifier to find a rare class of documents, such as hate speech directed at a particular religious group, requires labelled examples that are expensive to obtain when the class makes up only a small fraction of a corpus.

Extractive summary: sentences quoted from the sources.