Diagnosing Generalization Boundaries in AI-Based Vulnerability Detection via Calibrated Embedding Distances
This paper demonstrates that calibrated embedding geometry exposes these hidden generalization boundaries on syntactically similar vulnerable and patched code pairs.
ProofPaper ↗
Key points
- High benchmark scores in learning-based vulnerability detection mask severe degradation under distribution shifts.
- An evaluation of CodeBERT and Qwen2.5-Coder across pattern, project, and temporal shifts on a strictly paired dataset reveals three main findings.
- First, whitened intermediate-layer embeddings with local nearest-neighbour scoring separate unfamiliar samples with a mean out-of-distribution AUROC of 0.739 (RQ1).
- These calibrated distances establish a diagnostic signal that maps representation geometry directly to model degradation without requiring auxiliary models.
Sources (1)
- [1]Diagnosing Generalization Boundaries in AI-Based Vulnerability Detection via Calibrated Embedding DistancesarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 08:17 PM
This paper demonstrates that calibrated embedding geometry exposes these hidden generalization boundaries on syntactically similar vulnerable and patched code pairs.
High benchmark scores in learning-based vulnerability detection mask severe degradation under distribution shifts.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 9, 2026Qwen/Qwen-Image-2.1-Turbo
- Oct 9, 2026Introducing Clef-omni with full multimodality, plus a faster Clef and a cheaper Clef-flash
- Oct 8, 2026nvidia/Qwen3.8-Flash-Next-FP4-Local
- Oct 8, 2026ConwayResearch/Underdog-Saluki-27B-1.0
- Oct 8, 2026REMORY: Learning Residual Memory for Context Compaction
- Oct 8, 2026SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference