ReGRA: Residual-Gradient Adaptation for Complementary Low-Rank Fine-Tuning
Low-rank adaptation (LoRA) enables parameter-efficient fine-tuning by learning low-rank weight updates.
ProofPaper ↗
Key points
- Increasing rank expands adaptation capacity, but does not guarantee that all learned directions improve downstream performance.
- In a diagnostic experiment with rank-16 LoRA, we show that retaining only the top singular components achieves the best performance, whereas adding further components degrades it.
- Motivated by this, we propose Residual-Gradient Adaptation (ReGRA), a two-stage method for making more effective use of a fixed rank budget.
- An orthogonality regularizer encourages the two updates to capture complementary directions.
Sources (1)
- [1]ReGRA: Residual-Gradient Adaptation for Complementary Low-Rank Fine-TuningarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 07:23 AM
Low-rank adaptation (LoRA) enables parameter-efficient fine-tuning by learning low-rank weight updates.
Increasing rank expands adaptation capacity, but does not guarantee that all learned directions improve downstream performance.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 14, 2026Train LoRA directly on Quantized GGUFs, no PyTorch, no Parent(FP16) needed, just your GGUF and the data
- Oct 14, 2026Active Generative Simulation: Meta-Learning What to Simulate for Sim-and-Real Policy Co-Training
- Oct 13, 2026Comfy-Org/ComfyUI v0.40.0
- Oct 13, 2026DT-LoRA: Reaching Beyond the Tangent Space of Low-Rank Adaptation
- Oct 13, 2026Fish Audio Drama 3 Technical Report
- Oct 12, 2026UZHCL_for_the_win at Touche: Encoder Fine-Tuning, Synthetic Augmentation, and an Underpowered Encoder-LLM Comparison