CoTA++: Understanding the Repeat Curse in dMLLMs from an Information Flow Perspective
Diffusion-based multimodal large language models (dMLLMs) decode by iterative denoising with bidirectional attention, which precludes the standard autoregressive key--value cache.
ProofPaper ↗
Key points
- Practical deployments therefore rely on approximate caching that reuses intermediate states across denoising steps.
- Tracing the context-token information flow reveals three disruptions.
- Across layers, the deep-layer entropy of the context fails to converge where repetition arises.
- Across three dMLLMs and two caching backends, CoTA++ reduces the sample repetition rate of long responses from up to 99.6% to at most 17.0%, recovers most of the accuracy lost to caching on nine benchmarks, and keeps the speedup of caching at under 15% overhead, making approximate caching reliable for dMLLMs.
Sources (1)
- [1]CoTA++: Understanding the Repeat Curse in dMLLMs from an Information Flow PerspectivearXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 10:58 AM
Diffusion-based multimodal large language models (dMLLMs) decode by iterative denoising with bidirectional attention, which precludes the standard autoregressive key--value cache.
Practical deployments therefore rely on approximate caching that reuses intermediate states across denoising steps.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 16, 2026Welcome Beam: a 501B open model you can own, customize, and scale
- Oct 16, 2026DLite: History-free Speculative Decoding for Agentic AI
- Oct 16, 2026Xiaomi-Lens-0 Technical Report
- Oct 15, 2026Granite Switch 4.1: Chat, Decisions, and Speech from One Endpoint
- Oct 15, 2026huggingface/transformers v5.20.0: Release v5.20.0
- Oct 15, 2026When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill
