SITAR: A Situational Distortion Metric for Automatic Speech Recognition Evaluation

Key points

• In automatic speech recognition (ASR), transcription errors with similar lexical or semantic discrepancies can have markedly different consequences for the situation conveyed by speech.1
• Using targets derived from this formulation with limited offline LLM support, we train SITAR as a lightweight BERT-family regressor that scores reference-hypothesis pairs in a single forward pass, entirely free of any prohibitive runtime costs.1
• Recognizing that introducing a new metric demands rigorous justification, we formalize a principled validation protocol spanning both extrinsic validity, assessed at macro and micro levels, and intrinsic scorer reliability.1
• Through systematic validation, we demonstrate that SITAR holds strong potential as a practical, lightweight complement to existing ASR metrics.1

Sources (1)

• [1]SITAR: A Situational Distortion Metric for Automatic Speech Recognition Evaluation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 07:48 AM

  We introduce SITAR, a situational distortion metric for ASR, realized through a comprehensive end-to-end construction framework.

  In automatic speech recognition (ASR), transcription errors with similar lexical or semantic discrepancies can have markedly different consequences for the situation conveyed by speech.

Extractive summary: sentences quoted from the sources.