Dimension-Free PAC-Bayes Generalization for Deep Neural Networks via Dirichlet Processes

Key points

• For modern overparameterized neural networks where $D \gg n$, this parameter-counting dependency yields vacuous generalization bounds ($> 1{,}000%$).1
• In this work, we resolve this barrier by shifting prior specification from internal parameters to the non-parametric space of data distributions $\mathcal{M}1(\mathcal{Z})$ via a Dirichlet Process (DP), $\mathcal{DP}(α, F0)$, inducing parameter distributions through push-forward measures under the learning algorithm.1
• By the Data Processing Inequality, parameter-space relative entropy is strictly bounded by the data-space DP divergence, which we evaluate in exact closed form using Digamma functions.1
• Across vision architectures scaling up to 4.2 million parameters ($D = 7{,}850$ to $D = 4{,}203{,}522$) on the full MNIST benchmark, while classical weight-space bounds scale past $940%$ (becoming vacuous), our data-space DP-PAC-Bayes certificate remains stable, non-vacuous, and certified at $8.8%$--$9.4%$.1

Sources (1)

• [1]Dimension-Free PAC-Bayes Generalization for Deep Neural Networks via Dirichlet Processes
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 06:25 AM

  Classical PAC-Bayesian generalization theory places prior distributions over Euclidean parameter space $Θ\subseteq \mathbb{R}^D$, causing relative entropy to scale with network dimension $D$.

  For modern overparameterized neural networks where $D \gg n$, this parameter-counting dependency yields vacuous generalization bounds ($> 1{,}000%$).

Extractive summary: sentences quoted from the sources.