PACT: NVFP4 Data-Flow LoRA Fine-Tuning on Consumer GPUs
We present PACT, a LoRA fine-tuning system that accelerates frozen projections with native NVFP4 GEMMs while reducing quantization and dequantization overhead and limiting quantization error.
ProofPaper ↗
Key points
- Fine-tuning language models on consumer GPUs is constrained by computation and memory.
- To reduce activation memory, PACT reuses quantized RMSNorm outputs and saves gate/up activations in NVFP4 for backward.
- To limit numerical error, it preserves BF16 inputs for forward nonlinear operations and calibrates frozen weights offline using GPTQ adapted to NVFP4 scales and quantized activations.
- Evaluations across multiple models and tasks on a 16\,GB RTX 5080 show lower memory usage and faster LoRA fine-tuning with competitive accuracy, enabling model adaptation on resource-constrained consumer hardware.
Sources (1)
- [1]PACT: NVFP4 Data-Flow LoRA Fine-Tuning on Consumer GPUsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 18, 06:14 AM
We present PACT, a LoRA fine-tuning system that accelerates frozen projections with native NVFP4 GEMMs while reducing quantization and dequantization overhead and limiting quantization error.
Fine-tuning language models on consumer GPUs is constrained by computation and memory.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 17, 2026Nemotron 3.5 Bolt, Muse Spark Open, and a Fast LFM2.5 24B
- Oct 16, 2026Hugging Face ❤️ TorchTPU: TPUs, Natively
- Oct 16, 2026axolotl-ai-cloud/axolotl v0.21.0
- Oct 16, 2026Welcome Beam: a 501B open model you can own, customize, and scale
- Oct 15, 2026Noise-shaped Sampling and Annealing for Scalable On-the-edge Training of Binary and Ternary AI Models
- Oct 15, 2026NVIDIA Nemotron 3.5 Super VL Powers Specialized Agents with Multimodal Reasoning