Extreme Low-State Quantized Transformer for Electricity-Load Forecasting

Key points

• Aggressive low-state weight quantization can substantially reduce neural-network representational flexibility, making predictive performance difficult to preserve when only a few discrete weight states are available.1
• Quantization-aware training from random initialization (Scratch-QAT) also produced competitive low-state solutions, with particularly favorable behavior at three states.1
• We further introduce accumulated-update thresholding, which accumulates optimizer-proposed weight changes before commitment.1
• These results show that useful Transformer forecasting can be retained under highly discrete weight representations while strongly suppressing target-state activity, supporting a prospective route toward compact, low-activity edge forecasting using multilevel synaptic devices.1

Sources (1)

• [1]Extreme Low-State Quantized Transformer for Electricity-Load Forecasting
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 06:15 PM

  We investigate this regime for Transformer-based electricity-load forecasting using 3-, 5-, 7-, and 16-state weight quantization across feed-forward-only and attention-plus-feed-forward quantization.

  Aggressive low-state weight quantization can substantially reduce neural-network representational flexibility, making predictive performance difficult to preserve when only a few discrete weight states are available.

Extractive summary: sentences quoted from the sources.