NP-Hardness of Minimizing Neurons in Two-Hidden-Layer ReLU Neural Networks

Key points

• This paper resolves this question for two-hidden-layer ReLU networks under an $L^p(\mathbb{R}^d,\mathbb{R}^m)$ approximation constraint.1
• The result holds even when the target is represented by a rational ReLU network whose realization is nonzero, componentwise nonnegative, compactly supported, globally Lipschitz, and continuous piecewise affine.1
• The proof constructs compactly supported polyhedral frustum functions realized by two-hidden-layer ReLU networks and establishes the $L^p$-density of finite linear combinations of box-frustum functions.1
• The results offer theoretical justification for employing heuristic approximation methods in the design of ReLU neural networks, illustrating that attaining a minimal configuration within polynomial time is computationally unachievable.1

Sources (1)

• [1]NP-Hardness of Minimizing Neurons in Two-Hidden-Layer ReLU Neural Networks
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 06:19 AM

  A fundamental question in neural network architecture optimization is whether the minimum hidden-neuron count required to approximate a target function within a prescribed tolerance can be computed efficiently.

  This paper resolves this question for two-hidden-layer ReLU networks under an $L^p(\mathbb{R}^d,\mathbb{R}^m)$ approximation constraint.

Extractive summary: sentences quoted from the sources.