Geometric Calibration of Supervised Fine-Tuning
Within this view, supervised fine-tuning (SFT) is shown to induce dense updates that perturb the spectral geometry of pre-SFT weights along their principal directions, potentially disrupting the model's prior knowledge.
ProofPaper ↗
Key points
- Recent studies investigate the weight geometry through its spectral structure, which characterizes how the weights transform different input directions.
- These observations motivate using the geometry of pre-SFT weights as a reference for refining learned SFT updates.
- We study this perspective as Geometric Calibration and develop SPACE Subspace Projection for Adapted Checkpoint Enhancement), a lightweight post-hoc method that selectively reverts SFT update components within the principal subspace of the pre-SFT weights while preserving those in its orthogonal complement.
- These results highlight how the geometric relationship between SFT updates and pre-SFT weights shapes the adapted model's capabilities and, more broadly, motivate geometry-aware parameter updates for continual model improvement.
Sources (1)
- [1]Geometric Calibration of Supervised Fine-TuningarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 04:07 PM
Within this view, supervised fine-tuning (SFT) is shown to induce dense updates that perturb the spectral geometry of pre-SFT weights along their principal directions, potentially disrupting the model's prior knowledge.
Recent studies investigate the weight geometry through its spectral structure, which characterizes how the weights transform different input directions.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 13, 2026unslothai/unsloth v0.1.907-beta: Large Performance Gains + More Browser Use
- Oct 13, 2026Agent-Reflex: Teaching Language Agents to Act Reflectively
- Oct 13, 2026Cowork-Gym: Synthesizing Interactive Work Environments at Scale
- Oct 12, 2026U-Space: Uncovering When and Why Uncertainty Arises in Language Models [R][D]
- Oct 12, 2026Davinci-Tool: Scaling Agentic Tool-Use Supervision with Verifiable Executable Environments
- Oct 12, 2026TPPO: Tracking-Value PPO For Language Model Reasoning
