Weighted Ensemble Sampling for Rare-Event Estimation in Language Models

Key points

• Rare but severe failures of large language models can become practically significant at deployment scale.1
• WE samples exclusively from the original language model and improves upon direct Monte Carlo by periodically reallocating the sampling budget toward prefixes that appear more likely to reach the target rare event.1
• To guide this reallocation without learning a separate proposal distribution, we exploit internal LLM representations as prefix-level signals and empirically validate their effectiveness.1
• Experiments on safety-related tasks spanning both small-scale and modern large-scale LLMs demonstrate that WE improves the efficiency and reliability of rare-event probability estimation.1

Sources (1)

• [1]Weighted Ensemble Sampling for Rare-Event Estimation in Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 09:11 PM

  We introduce Weighted Ensemble (WE), a training-free framework for rare-event estimation in language models.

  Rare but severe failures of large language models can become practically significant at deployment scale.

Extractive summary: sentences quoted from the sources.