S-EDL: Eliciting Self-Evidence from Sequence Likelihoods for Semantic Calibration of LLMs

Key points

• Calibrating large language models (LLMs) in open-ended generation is uniquely challenging because many distinct token sequences can express the same underlying meaning.1
• Prior calibration techniques target either fixed-set classification confidence or the binary correctness of individual strings, making them inherently ill-suited for the dynamic, multi-string nature of open-ended semantic outcomes.1
• By eliciting evidence from the sequence likelihoods of the model itself, S-EDL constructs a differentiable Dirichlet prior over prompt-specific semantic classes.1
• By aligning generative likelihoods with semantic correctness, this work establishes a framework for deploying trustworthy LLMs in complex, open-ended applications.1

Sources (1)

• [1]S-EDL: Eliciting Self-Evidence from Sequence Likelihoods for Semantic Calibration of LLMs
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 02:55 PM

  To bridge this gap, we introduce S-EDL, a novel evidential method that natively optimizes LLMs for semantic calibration.

  Calibrating large language models (LLMs) in open-ended generation is uniquely challenging because many distinct token sequences can express the same underlying meaning.

Extractive summary: sentences quoted from the sources.