Label Collapse in Diffusion Language Model In-Context Learning
For autoregressive models, collapse persists where the majority-label-bias account does not apply, and arrangement patterns can reverse the collapse direction; for diffusion models, we present a systematic evaluation of ICL failure modes, complementing contemporaneous studies of dLLM context comprehension and demonstration-order sensitivity, and identify open questions about how iterative denoising interacts with label biases.
ProofPaper ↗
Key points
- In-context learning (ICL) applies large language models to new tasks by prepending a few labeled demonstrations to a query.
- Prior work has studied how demonstration ordering affects ICL, but the interaction between label arrangement patterns and model biases has not been systematically examined.
- Critically, accuracy alone cannot detect this collapse: models can achieve near-chance accuracy while generating query-independent labels, a score indistinguishable from balanced performance, so the failure can persist undetected in deployment.
- We measure separation, the log-odds of the two label words at the query position (the quantity that label-bias calibration operates on), as a complementary diagnostic that exposes collapse hidden beneath surface accuracy, together with a measurement protocol that guards label extraction against tokenizer and formatting artifacts.
Sources (1)
- [1]Label Collapse in Diffusion Language Model In-Context LearningarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 02:17 PM
For autoregressive models, collapse persists where the majority-label-bias account does not apply, and arrangement patterns can reverse the collapse direction; for diffusion models, we present a systematic evaluation of ICL failure modes, complementing contemporaneous studies of dLLM context comprehension and demonstration-order sensitivity, and identify open questions about how iterative denoisin
In-context learning (ICL) applies large language models to new tasks by prepending a few labeled demonstrations to a query.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 11, 2026TabMosaic: Unified In-Context Diffusion for Tabular Generative Tasks
- Oct 11, 2026autotrust/JEV-35B-NVFP4
- Oct 10, 2026Toward System One Clinical Intelligence: Parameter-Efficient Fine-Tuning and Counterfactual Explainability of Decision Foundation Models
- Oct 10, 2026vllm-sr/d3
- Oct 10, 2026CB-Agent: Concept-Based In-Context Learning for Sequential Action Selection
- Oct 9, 2026Microsoft's Decision-1 model enters the fast-growing AI decision model race