Quantization can compress large language models to extreme low bit widths, but the resulting models often lose their ability to perform long-form reasoning. A paper submitted to arXiv on September 22, 2026, identifies the root cause and proposes a targeted fix. Quantization-aware distillation, the standard recovery method, restores short-form question-answering performance but leaves mathematical derivations and code generation substantially impaired. Long generations degenerate into repetitive loops that exhaust the decoding budget without completing a solution.
The paper, from Yuanteng Chen, Zhilei Liu, Peisong Wang, and colleagues at the Chinese Academy of Sciences and City University of Hong Kong, introduces a two-stage framework. The first stage uses quantization-aware distillation to provide a stable low-bit initialization. The second stage, called on-policy distillation, places teacher supervision on trajectories the quantized model actually generates during deployment. Across four models at 2.79 and 1.88 effective bits, the combined pipeline doubles average performance retention on MATH-500 from 35 percent to 70 percent and raises HumanEval retention from 66 percent to 91 percent, all while preserving the short-form capabilities recovered by the first stage.
The Problem: QAD Recovers Answers but Not Reasoning
At four bits and above, post-training quantization preserves performance with only a small calibration set. Below three bits, quantization errors become harder to compensate for, and quantization-aware distillation becomes essential. QAD trains a quantized student to match a full-precision teacher on a manageable distillation corpus. The teacher provides supervision on prefixes drawn from fixed reference responses.
The recovery, however, is strikingly uneven. Across four models and two effective bit widths, QAD retains an average of 86 percent of the BF16 reference performance on short-form question answering benchmarks that require no autoregressive generation. Retention drops to 56 percent on GSM8K and 59 percent on MBPP, then falls further to 27 percent on MATH-500 and 16 percent on AMC23. What distinguishes these bands is how much the model must generate for itself. Benchmarks that score answer options by likelihood require no generation. Benchmarks requiring extended reasoning demand hundreds or thousands of tokens.
On MATH-500, 95 percent of QAD generations exhaust the decoding budget, compared with 27 percent for BF16. Moreover, 70 percent of QAD outputs end in repeated 8-grams, against 12 percent for BF16. The model continues producing tokens but its reasoning becomes trapped in repetitions that prevent it from reaching a conclusion. The recovery deficit grows where the student must sustain a longer chain of its own predictions.
Quantization-Amplified Exposure Bias
The root cause is a mismatch amplified by quantization. Teacher-forced QAD trains the quantized student to match its teacher on prefixes drawn from a fixed corpus. During deployment, the student conditions on its own previous predictions. Quantization perturbs the next-token distribution at every step, and each departure changes the context for subsequent predictions.
Even a locally plausible continuation can move the model away from the trajectories covered during training. Further predictions then combine quantization error with the effects of that altered context, allowing deviations to accumulate along the sequence. Lower precision increases the disruption, and longer generations provide more opportunities for it to compound. This mechanism predicts a breakdown that unfolds during generation: as deviations accumulate, responses become harder to terminate and increasingly prone to repetition.
This is exposure bias amplified by quantization. The missing guidance lies along the reasoning trajectories the quantized model actually generates. QAD teaches the student how to continue demonstrated trajectories, but the model must complete its reasoning from prefixes produced by its own perturbed predictions.
On-Policy Distillation: Supervision Where the Model Goes
The solution is to shift teacher supervision from fixed corpus prefixes to the student's own trajectories. The paper introduces on-policy distillation as a second stage that starts from a QAD checkpoint.
In teacher-forced QAD, the loss minimizes the KL divergence between the teacher distribution and the student distribution evaluated on reference prefixes from a fixed corpus. In on-policy distillation, the student samples completions through the quantized forward path used at deployment. A frozen BF16 teacher provides token-level feedback on the prefixes the student actually produces. Sampling through the low-bit path makes quantization-induced changes in the student's behavior part of the training distribution itself.
OPD combines two complementary signals. The first is a reverse KL term that supplies token-level feedback on the prefixes the student generates, penalizing continuations the student favors but the teacher assigns low probability to. The second is a group-relative advantage term that promotes completions reaching verified solutions. A verifier assigns each completion a reward: final-answer correctness for mathematics and test execution for code.
The training uses the reverse-KL loss from the student's own samples combined with an advantage-weighted policy gradient. The two signals guide recovery at complementary scales: the sampled reverse-KL term provides dense token-level guidance throughout each completion, while the group-relative advantage promotes completions that reach verified solutions.
Experimental Results
Experiments evaluate four models: Qwen3-0.6B, Qwen3-1.7B, Qwen3-4B, and Falcon3-1B-Instruct, quantized to 2.79 and 1.88 effective bits using round-to-nearest quantization with 8-bit activations and 4-bit embedding and output-head weights. All OPD runs start from the reported QAD checkpoint, ensuring a common starting point. Recovery proceeds in a mathematics phase followed by a code phase.
On MATH-500, OPD raises average BF16 retention from 35.3 percent to 69.7 percent. On GSM8K, retention improves from 62.7 percent to 85.0 percent. Across all six Qwen3 configurations, GSM8K accuracy improves by 8.4 to 47.2 percentage points. Code generation improves alongside mathematical reasoning. At 2.79 bits, Qwen3-1.7B raises MBPP from 35.3 percent to 52.0 percent and HumanEval from 41.5 percent to 59.1 percent, approaching the BF16 scores of 54.0 percent and 67.1 percent.
Broad capabilities are preserved. Mean QA9 retention increases from 88.1 percent to 92.1 percent, so the combined pipeline restores reasoning while maintaining short-form performance. OPD's contribution grows as precision falls. For Qwen3-4B at 1.88 bits, OPD raises GSM8K accuracy from 17.36 percent to 64.59 percent and MBPP from 11.6 percent to 48.7 percent.
OPD is dramatically more efficient than continued teacher-forcing. In a controlled comparison under matched budgets, OPD outperforms continued QAD in all 16 comparisons across four models and bit widths. For Qwen3-1.7B at 2.79 bits, OPD nearly doubles the MATH-500 accuracy of continued QAD, reaching 45.6 percent versus 23.2 percent. Extending the QAD mathematics phase to two or three times the matched budget yields essentially no further gain. OPD converts training steps into reasoning gains far more efficiently, yielding up to 42 times more GSM8K points per thousand optimizer steps. The OPD stage uses roughly 14 to 23 times fewer GPU-hours than the QAD initialization stage.
Against existing baselines, OPD reaches 84 percent retention on MATH-500 at 2.79 bits while the strongest competing methods retain at most 59 percent. Below two bits, every post-training and quantization-aware baseline scores zero on MBPP and HumanEval, while OPD reaches 64 percent on GSM8K and 77 percent on MBPP.
How OPD Changes Generation Behavior
The behavioral recovery is as striking as the accuracy gains. On MATH-500, OPD reduces the loop rate from 70 percent to 17 percent, close to the BF16 reference of 12 percent, and cuts budget exhaustion from 95 percent to 53 percent. On GSM8K, the loop rate falls from 30 percent to 2 percent and budget exhaustion from 32 percent to 5 percent, matching the BF16 reference on both measures.
The larger improvement on MATH-500 mirrors the greater disruption on longer generations. OPD recovers more ground where QAD's trajectory failures are most severe. By extending teacher guidance onto the student's own prefixes, OPD helps the low-bit model sustain a derivation and bring it to a conclusion.
Limitations and Open Questions
The paper evaluates on Qwen3 and Falcon3 architectures with round-to-nearest quantization. The approach has not been validated on other model families, quantization schemes such as mixed-precision or vector quantization, or on tasks beyond mathematical reasoning and code generation. Whether the framework transfers to reinforcement-learning-trained reasoning models remains open.
The OPD stage requires a frozen BF16 teacher, which adds memory overhead. The teacher must be available during both training and generation. For deployment settings where memory is severely constrained, maintaining the full-precision teacher may be impractical. The paper does not explore whether a smaller distilled teacher could substitute.
The training requires two phases with carefully selected corpora. The mathematics phase uses GSM8K, MATH, and DAPO-Math; the code phase uses MBPP and KodCode. Adapting the approach to different task domains would require identifying appropriate training corpora and verifiers. The reliance on task verifiers, particularly for code generation where test execution is required, adds infrastructure complexity.
Finally, the experiments use non-thinking mode at temperature 1. Whether OPD's benefits transfer to thinking-mode generation, where the model produces longer chain-of-thought sequences, is an interesting direction for future work.
What This Means for Practitioners
For teams deploying quantized models on resource-constrained hardware, this paper offers a practical pipeline. The QAD stage provides a stable low-bit initialization that preserves broad capabilities. The OPD stage then recovers reasoning with a few hundred additional optimizer steps, requiring substantially less compute than extended QAD training. The code is publicly available.
The central insight, that exposure bias is amplified by quantization, extends beyond this specific framework. Any method that trains quantized models on fixed reference trajectories while deploying them on their own generated sequences will face this mismatch. On-policy distillation addresses it by aligning the training distribution with the deployment distribution. This principle may generalize to other scenarios where quantization and autoregressive generation interact.
For the broader field of model compression, the results reframe the trade-off between compression and capability. Extreme quantization below three bits does not merely degrade performance uniformly; it breaks specific reasoning behaviors that depend on sustained autoregressive generation. Targeted recovery of these behaviors, rather than generic capability restoration, is essential for deploying low-bit models on tasks requiring extended reasoning.