Multi-teacher on-policy distillation aims to combine the strengths of RL-trained teachers in a single student, but how teacher signals affect parameter changes remains underexplored. Researchers studied Qwen3-1.7B with four domain teachers trained with RL from the same initialization as the student, comparing gradients, optimizer updates, and task learning curves, with additional SmolLM3-3B diagnostics.
Loss Averaging Implicitly Weights Responses
Token averaging favors longer responses, and equalizing domain contributions retains this weighting within domains. This means the choice of averaging strategy directly shapes which teacher signals dominate the student's parameter updates.
Adam's First Moment Reduces Differences in Parameter Updates
The cosine similarity between teachers measures 0.83, while similarity between averaging rules reaches 0.96, despite raw gradient differences. Adam's momentum mechanism therefore compresses the distance between distinct teacher signals, making updates more consistent across domains.
BF16 Rounding Hides Small Changes
About 97% of FP32 master weights differ from initialization, but only 7–11% of BF16 weights do. This quantization effect means that low-precision training can mask the true magnitude of parameter movement during distillation.
Top-64 Intersection KL Gradient Matches Full-Vocabulary Gradient
The top-64 intersection KL gradient closely matches Qwen's full-vocabulary gradient, yet the effect on task performance depends on averaging. Mathematics accuracy is 2.6 points higher than with sampled-token policy-gradient under response averaging and 2.1 points lower under global token averaging.
These findings clarify why multi-teacher distillation behaves as it does and offer practical guidance for practitioners selecting averaging strategies and precision formats.