Noisy-Label Calibration with Multiple Unreliable Labels

Key points

• In medical annotation and crowdsourcing, multiple annotators assign labels to each example, but these labels may be incorrect.1
• Treating such noisy labels as ground truth can bias calibration evaluation.1
• To address this problem, we propose methods for evaluating and recalibrating a fixed binary predictor under the underlying clean-label distribution, using only noisy annotations.1
• Then, we propose practical algorithms for estimating unknown noise probabilities under conditional independence and label-orientation assumptions.1

Sources (1)

• [1]Noisy-Label Calibration with Multiple Unreliable Labels
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 04:02 AM

  First, given annotator-specific noise probabilities, we propose Noise Corrected ECE (NC-ECE), a binned estimator of expected calibration error (ECE) under class-conditional noise.

  In medical annotation and crowdsourcing, multiple annotators assign labels to each example, but these labels may be incorrect.

Extractive summary: sentences quoted from the sources.