What Matters for NVFP4 Training? A Comparative Study of Low-Precision Pretraining Recipes

Key points

• Training large language models directly in 4-bit floating-point (FP) formats promises substantial improvements in throughput and energy efficiency.1
• While some recipe design choices have been validated at scale, many promising approaches remain untested beyond small models and short token horizons, leaving open the question of which trends will hold.1
• We propose a final recipe grounded in the principles behind established stable NVFP4 training at scale, incorporating state-of-the-art techniques such as unbiased gradient estimation with lower quantization error than stochastic rounding.1
• Through ablation studies, we find that: (i) each technique in the optimized recipe measurably improves loss trajectory, (ii) selective high-precision layers are necessary for recovering accuracy at scale, (iii) not all tensors in the backward pass benefit equally from de-biasing, leaving room to apply complementary error-reduction techniques to the remaining tensors.1

Sources (1)

• [1]What Matters for NVFP4 Training? A Comparative Study of Low-Precision Pretraining Recipes
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 17, 12:23 AM

  We present a systematic comparison of recent NVFP4 recipes at medium model scale, evaluating dense 8B and 30B-A3B MoE models trained for up to 1T tokens, with a focus on which ingredients are necessary for accuracy recovery.

  Training large language models directly in 4-bit floating-point (FP) formats promises substantial improvements in throughput and energy efficiency.

Extractive summary: sentences quoted from the sources.