Towards Universal Calibration for In-Context Classification

Key points

• In-Context Learning (ICL) has received a lot of interest recently due to its ability to improve classification accuracy for new tasks without having to fine-tune the model over these tasks.1
• Recent work has shown that ICL can have a negative impact on calibration due to biased selection of in-context examples.1
• Specifically, we design a transformer-based calibrator which uses the model predictions on in-context examples to predict a confidence score for the test input.1
• We show that the calibrator trained on three models and ten tasks, is able to generalize/calibrate ICL predictions on several new tasks and four LLMs unseen during training.1

Sources (1)

• [1]Towards Universal Calibration for In-Context Classification
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 04:55 PM

  In this work, we introduce CALIBFORMER, a universal ICL calibrator for classification tasks, that generalizes to new models and tasks without further fine-tuning on these tasks.

  In-Context Learning (ICL) has received a lot of interest recently due to its ability to improve classification accuracy for new tasks without having to fine-tune the model over these tasks.

Extractive summary: sentences quoted from the sources.