Right On Q: Recovering Reasoning under Quantization via Self-Generated Trajectories
Quantization-aware training (QAT) aims to recover reasoning capabilities in low-bit large language models (LLMs).
ProofPaper ↗
Key points
- Existing recipes typically supervise the quantized student with the original full-precision model on fixed response sequences.
- However, the resulting low-bit quantized models still struggle on long-horizon reasoning tasks and often fail to terminate within a token budget that suffices for the full-precision model.
- We present OnQ, a low-bit recovery pipeline that combines on-policy distillation and candidate-wise Dynamic KL.
- Dynamic KL weights forward and reverse KL by each candidate's teacher-student probability ratio to balance correction of under- and overprediction.
Sources (1)
- [1]Right On Q: Recovering Reasoning under Quantization via Self-Generated TrajectoriesarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 08:03 AM
Quantization-aware training (QAT) aims to recover reasoning capabilities in low-bit large language models (LLMs).
Existing recipes typically supervise the quantized student with the original full-precision model on fixed response sequences.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 20, 2026Where Do Legal Factors Live in LLMs?
- Oct 20, 2026BitMTP: When Multi-Token Prediction Meets Low-Bit Large Language Models
- Oct 20, 2026Do Diffusion LLMs Scale? Testing Mercury 3 + Decide
- Oct 20, 2026MPS in PyTorch 2.15
- Oct 19, 2026Safety Mid-Training: Internalizing LLM Safety as a Foundational Capability
- Oct 19, 2026Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM