Zeroth-Order Fine-Tuning of LLMs in Learnable Low-Rank Subspaces
Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models (LLMs).
ProofPaper ↗
Key points
- We propose AdaSub-ZO, a novel forward-only fine-tuning framework in learnable low-rank subspaces.
- Instead of relying on fixed random matrices, AdaSub-ZO iteratively learns layer-wise projection matrices directly from finite-difference feedback to track task-aware descent directions.
- Furthermore, AdaSub-ZO incorporates a subspace momentum acceleration mechanism to smooth noisy gradient estimates across evolving subspace bases.
- Extensive experiments on natural language understanding tasks demonstrate that AdaSub-ZO outperforms existing ZO baselines, substantially reducing gradient estimation variance and accelerating convergence.
Sources (1)
- [1]Zeroth-Order Fine-Tuning of LLMs in Learnable Low-Rank SubspacesarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 10, 11:33 PM
Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models (LLMs).
We propose AdaSub-ZO, a novel forward-only fine-tuning framework in learnable low-rank subspaces.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 10, 2026SocraticRepair: Planning-Guided Socratic Tutoring for Program Debugging and Misconception Repair
- Oct 10, 2026Toward System One Clinical Intelligence: Parameter-Efficient Fine-Tuning and Counterfactual Explainability of Decision Foundation Models
- Oct 10, 2026Training-Free BioNEL-RAG: Advancing Biomedical Entity Linking through Retrieval-Augmented Generation and Large Language Model
- Oct 10, 2026Vela: User Behavior as Supervision Sparks LLM Intelligence
- Oct 9, 2026Minions: A Dynamic and Distributed Virtual GPU Cloud Built on Idle Apple Silicon Devices
- Oct 8, 2026Understanding Jev, the new model everyone is talking about