When Sparsity Breaks Connectivity: Functional Reachability and Guided Topology Dynamics in Extreme Sparse Training

Key points

• We formalize forward, backward, and functional reachability on layered sparse graphs, derive closed-form expectations for reachable neurons and edges, and show that Monte Carlo realizations match the theory.1
• We introduce this reachability-driven modification of topology evolution as Reachability-guided in DST, referred to as guided DST.1
• On CIFAR-10/100 and Tiny ImageNet, guided DST recovers large accuracy gaps versus unguided DST (up to $+14.4$ points at $99.9%$ sparsity on CIFAR-100), and on MLP-Mixer guided DST remains trainable at $99.99%$ sparsity while unguided random regrowth degrades sharply.1
• These results cast extreme-sparse training as a connectivity-preserving dynamical process.1

Sources (1)

• [1]When Sparsity Breaks Connectivity: Functional Reachability and Guided Topology Dynamics in Extreme Sparse Training
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 09:38 PM

  Dynamic Sparse Training (DST) periodically prunes and regrows connections, inducing a discrete dynamical mechanism over network topologies.

  We formalize forward, backward, and functional reachability on layered sparse graphs, derive closed-form expectations for reachable neurons and edges, and show that Monte Carlo realizations match the theory.

Extractive summary: sentences quoted from the sources.