Compute-Optimal Scaling Laws for Spiking Language Models

Key points

• Spiking neural networks target energy-efficient language modeling through sparse, binary activations, but the efficiency-accuracy tradeoff of spiking language models is uncharacterized at scale.1
• First, the compute-optimal frontier is $N{opt} \propto C^{0.56}$, close to the 0.50 that Hoffmann et al. report for dense Transformers, so spiking does not change how parameters and data should be traded off under a compute budget.1
• Second, against non-spiking RWKV and Transformer baselines trained the same way, the loss gap splits into a backbone cost of about 0.16 nats that does not change with compute and a spiking cost that falls from 0.34 to about 0.23 nats and then levels off.1
• These results give the first compute-optimal recipe for spiking language models and quantify the accuracy cost of their sparsity.1

Sources (1)

• [1]Compute-Optimal Scaling Laws for Spiking Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 10, 07:46 PM

  We present the first Chinchilla-style IsoFLOP scaling study of a spiking language model, sweeping SpikeGPT-family models (RWKV backbone) from width 384 to 1536 across compute budgets from $3\times10^{17}$ to $10^{19}$ FLOPs on FineWeb-EDU.

  Spiking neural networks target energy-efficient language modeling through sparse, binary activations, but the efficiency-accuracy tradeoff of spiking language models is uncharacterized at scale.

Extractive summary: sentences quoted from the sources.