When Your Teacher Becomes the Constraint
Training language model agents on multi-turn tasks with reinforcement learning has a fundamental sparse-reward problem. An agent interacts with an environment over many steps, but receives only a single scalar reward at the end of the trajectory. Every intermediate decision, every tool call, every state transition goes unsupervised. On-policy distillation addresses this by having a teacher provide dense, token-level feedback on trajectories the student actually generates. The standard recipe uses a skill-conditioned teacher, the same model prompted with privileged task information like retrieved skills, and trains the student to reproduce the teacher's behavior without that context. The skills get internalized into the student's parameters, and no extra context is needed at inference.
This recipe rests on two assumptions that Yan Yu, Zhengxi Lu, and colleagues at Zhejiang University and Alibaba Group show do not hold in practice. Privileged information alone does not make a teacher reliable. And the benefit of teacher supervision is stage-dependent, useful early but harmful later. Their solution, RetireOPD, trains a competent teacher first, uses it to bootstrap the student, and then lets the student fire the teacher when the supervision stops helping.
Two Assumptions That Break Down
The standard approach to on-policy self-distillation (OPSD) keeps the teacher and student as two branches of one shared policy. The teacher branch receives privileged skill context; the student branch does not. Both are trained jointly, and the student learns to mimic the teacher. The problem is that the shared parameters are optimized mainly through the student's skill-free objective, so the policy is never explicitly trained to act on skill context.
The paper's experiments on ALFWorld reveal the consequences. A Qwen2.5-7B model prompted with the same skills reaches only 23.4% success rate, worse than the skill-free student. The 3B model reaches 28.9%. Simply having access to extra information does not translate into better behavior unless the model is explicitly trained to use it. The teacher must be trained to exploit its privileged context before it can supervise anyone.
The second assumption fails over time. Adding on-policy distillation to GRPO (Group Relative Policy Optimization) removes the slow start of pure GRPO and the low ceiling of pure OPD. But the teacher-student discrepancy first narrows and then widens. Early on, the student absorbs the teacher's behavior and the gap closes. Later, as the student improves through reward optimization, it discovers actions the teacher does not take. The two objectives begin to conflict. Continuing to match the teacher beyond this point holds the student near the teacher's performance ceiling rather than letting it surpass it.
A first-order analysis in the appendix shows that a stagnating discrepancy implies locally opposed gradients. The GRPO gradient pushes toward higher reward; the OPD gradient pulls toward the teacher's distribution. When these disagree, the student cannot improve on either front. Teacher supervision is temporary scaffolding, and the question is when to remove it.
RetireOPD: Three Stages
RetireOPD has three stages that address these findings directly.
Stage 1: Teacher Construction. A skill-conditioned teacher is trained separately with environment rewards under GRPO. This is the key architectural decision that distinguishes RetireOPD from prior work. Instead of using the same model for both teacher and student, the teacher is decoupled and optimized specifically to exploit the privileged skill context. The teacher objective maximizes expected reward given the task input and the skill context c+. After training, the teacher is frozen and used only for distillation.
This step converts the information advantage of privileged context into a behavioral advantage. Without it, the teacher is unreliable. With it, the 3B teacher reaches 79.7% on ALFWorld and the 7B teacher reaches 90.6%, compared with 28.9% and 23.4% for unoptimized skill-prompted models.
Stage 2: Joint GRPO-OPD Training. The student is trained without access to skill context using a combined objective: the GRPO loss plus a weighted OPD loss. GRPO provides trajectory-level reward signals using group-relative advantages computed from G sampled trajectories. OPD provides dense token-level supervision by comparing the student's distribution against the frozen teacher's distribution at each position in student-generated trajectories.
The OPD loss uses a sampled-token approximation of the reverse KL divergence: at each position t, the token-level gap is the difference between the teacher's log-probability and the student's log-probability for the token that the student actually generated. The negation of this gap estimates the reverse KL at that position. This avoids summing over the full vocabulary at every step.
Stage 3: Adaptive Teacher Retirement. Training is divided into monitoring windows of W steps. At each window, two signals are computed. The alignment progress measures whether the teacher-student discrepancy is still decreasing: a negative value means the gap is shrinking; a non-negative value means it has stalled or reversed. The relative competence measures the student's success rate as a fraction of the teacher's, averaged over two consecutive windows to reduce noise.
The teacher is retired at the first window where both conditions hold: discrepancy stagnation and sufficient competence. The default thresholds are delta equals zero and gamma equals 0.9. After retirement, OPD is removed and training continues with GRPO alone. No further teacher forward passes are required.
Results: Surpassing the Teacher
Across Qwen2.5 models from 1.5B to 7B on ALFWorld and WebShop, RetireOPD consistently outperforms all baselines and surpasses its own skill-conditioned teacher in every setting.
On ALFWorld, RetireOPD improves over the GRPO baseline by 18.8 points on the 3B model (93.8% vs. 75.0%), with gains of 17.0 and 14.1 points on the 1.5B and 7B models respectively. On WebShop, the improvements are 19.0 points on the 1.5B model (75.8% vs. 56.8%) and 14.0 and 11.8 points on the 3B and 7B models.
The comparison with the teacher is particularly striking. On Qwen2.5-3B, the skill-conditioned teacher achieves 79.7% on ALFWorld and 64.8% accuracy on WebShop. RetireOPD's student reaches 93.8% and 77.3%. The student does not merely match the teacher; it exceeds it. This happens because retirement removes the constraint that was holding the student back. After the teacher is dropped, GRPO continues to optimize for reward without the competing pull of distillation, and the student finds actions the teacher never explored.
The training dynamics confirm this story. Under continued joint optimization (GRPO+OPD without retirement), the success rate plateaus while the teacher-student discrepancy keeps widening. Switching to OPD alone reduces the gap but fails to improve task performance. Removing OPD and continuing with GRPO steadily improves the success rate from 76.6% at the retirement point to 93.8%.
Ablations: Both Signals Matter
The ablation studies isolate the contribution of each component. Without retirement (keeping OPD throughout training), the 3B model reaches only 82.8% on ALFWorld, 9.4 points below the full method. Using only the alignment stagnation signal (no competence threshold) reaches 89.0%, and using only the competence signal (no stagnation check) also reaches 89.0%. Combining both achieves 92.2%, showing they are complementary.
The retirement thresholds are robust. Varying gamma from 0.80 to 0.96 and delta from -0.10 to 0.04 shifts the retirement step from 50 to 95, but success rates remain between 89.1% and 92.2%. The retiring step stays within plus or minus 5 steps of the default for a broad parameter range. This is important for practical use: the method does not require precise threshold calibration.
Comparing adaptive retirement with linear annealing (the ATOD strategy of decaying the distillation weight over a fixed schedule) shows that the adaptive approach continues improving after the annealing approach plateaus. Teacher utility does not decay according to a predefined schedule, and the training signal provides better guidance for when to transition.
Why This Matters for Agent Training
The practical implication is straightforward. If you are training a multi-turn agent with RL and on-policy distillation, do not assume your teacher is reliable just because it has privileged information. Train the teacher separately with environment rewards first. And do not assume distillation helps throughout training. Monitor the teacher-student discrepancy and the student's relative competence, and retire the teacher when both signals indicate the supervision has become a constraint.
The code is available at https://github.com/ZJU-REAL/SDAR. The method is implemented on top of the SDAR framework and uses the SkillBank from SkillRL as privileged information, with keyword matching for skill retrieval. The retirement evaluation runs every 5 steps by default.
RetireOPD illustrates a broader principle in agent training: scaffolding that helps early can hurt later, and the transition point depends on the student's learning dynamics rather than a fixed schedule. The discrepancy between teacher and student is not a defect to be minimized throughout training. It is a signal that, once it stops shrinking, tells you the teacher has done its job. The student is ready to operate on its own.