Conducting Reliable Keyword Estimation with Unreliable ASR in Police Audio Data

Key points

• Extracting insights from large audio data sets using human transcription is expensive and time-consuming.1
• Off-the-shelf Automatic Speech Recognition (ASR) systems offer an easy-to-use scalable alternative, but systematic transcription errors, especially in domain-specific settings, can bias downstream analyses and lead to unreliable conclusions.1
• We evaluate this approach on police radio communications from Baltimore and Chicago, a domain in which keyword distributions could provide valuable insights into police activity, but off-the-shelf ASR systems exhibit systematic errors on operationally important terminology.1
• Using bootstrap-based confidence interval coverage as our evaluation metric, we find that our approach substantially increases the number of domain-specific keywords that can be analyzed with statistically valid confidence intervals, from 43.0% using ASR alone and 27.2% using only a small set of human-labeled transcriptions to 56.2% in the Baltimore dataset, with similar improvements observed in Chicago data.1

Sources (1)

• [1]Conducting Reliable Keyword Estimation with Unreliable ASR in Police Audio Data
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 03:48 PM

  We investigate whether imperfect ASR transcripts can still be used to obtain statistically valid keyword frequency estimates by combining model outputs with a small set of human-labeled examples, following Prediction Powered Inference and Design-based Supervised Learning frameworks.

  Extracting insights from large audio data sets using human transcription is expensive and time-consuming.

Extractive summary: sentences quoted from the sources.