Beyond 4 bits: Data-Centric Post-training Quantization and Recovery

Key points

• Down to 4 bits it is near-lossless, but at 2 bits scalar PTQ configurations fall to chance-level accuracy.1
• We study three choices usually treated as implementation details that matter little at 4 bits but decide 2-bit quality: the quantization representation, the calibration data, and the recovery fine-tuning data, across six dense and mixture-of experts models (ranging from $3.8$ to $32$B parameters).1
• Representation is the largest factor and vector codebooks restore usable models where scalar PTQ pipelines collapse.1
• Including codebooks, it uses about $2.3$ effective bits per weight and retains $89$ to $97%$ of full-precision question-answering (QA) accuracy where scalar PTQ is at chance, and, including recovery, uses $1.4$ to $30\times$ less compute than the strongest quantization-aware-training (QAT) baseline.1

Sources (1)

• [1]Beyond 4 bits: Data-Centric Post-training Quantization and Recovery
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 05:10 PM

  Post-training quantization (PTQ) compresses the weights of a trained model with limited compute and data.

  Down to 4 bits it is near-lossless, but at 2 bits scalar PTQ configurations fall to chance-level accuracy.

Extractive summary: sentences quoted from the sources.