The problem this paper addresses is fundamental to how agentic on-policy distillation (OPD) trains student agents. In OPD, a teacher model scores every token generated by a student, and the student learns from these token-level scores. A natural explanation for where OPD fails is compounding error: the student's mistakes push its trajectories beyond the region where the teacher is reliable, so recent methods keep training inside that region through episode-length curricula, teacher-guided turns, prefix replay, or teacher intervention where the two disagree. The authors show this explanation is incomplete.
From the states where the student departs most from the teacher, the teacher's own takeovers still finish the task, yet the student trained on its token scores keeps running out of turns. The authors attribute this to the myopia of next-token scoring. The teacher rates each token without seeing where the episode ends, so its scores let through the safe, delaying moves that outcome-based RL would punish with zero reward, a penalty agentic OPD lacks. The authors call this difference between what the teacher does and what it scores the rollout-scorer gap.
Rollout-to-Scorer Distillation
The core proposal is Rollout-to-Scorer Distillation (R2S), which repairs the scorer before distillation. The procedure has four steps. First, mine high-disagreement states from the untrained student—these are states where the student's actions diverge most from the teacher's. Second, run teacher takeovers from those states; the teacher generates continuation sequences that may succeed even when the student's original trajectory would fail. Third, keep only the states where enough takeovers pass a reward-free check such as consensus or an LLM judge. Fourth, fine-tune the teacher to give its takeover-informed scores on every assistant token from the divergence turn onward, without the takeover in context. The repaired teacher is then frozen and serves as the sole scorer for standard OPD, requiring no task reward during repair or training.
Why next-token scoring is myopic
Next-token scoring evaluates each assistant token independently, without considering whether the episode will ultimately succeed or fail. This creates a mismatch: the teacher may assign high probability to a delaying move that preserves options later, but outcome-based RL would assign zero reward if that move prevents task completion. In agentic OPD, the student receives these myopic scores and learns behaviors that look token-wise reasonable but accumulate into failure. The rollout-scorer gap is the measurable distance between the teacher's token scores and the actual outcome of the rollout.
Empirical results across environments
The authors evaluate R2S across three environments, two teachers, and four students, comparing against three state-of-the-art agentic-OPD methods. In all twelve teacher-student-environment cells, R2S outperforms the best competing method by 12.8 success points on average. The improvement is consistent: R2S exceeds the baseline in every cell, with no environment or student model where it underperforms. The authors also ablate the repair step, showing that fine-tuning the teacher on takeover-informed scores is essential—removing it collapses performance to the untrained baseline.
Handling memory-dependent teacher responses
One technical detail the authors explore is how to handle teacher responses that depend on external memory unavailable to the student. In the RPent harness, the teacher uses a memory module that the student does not have access to at test time. The authors compare two supervision strategies: masking out memory-dependent responses, and rewriting them before retention. The rewrite-and-filter approach, which rewrites teacher reasoning and tool-call arguments and keeps only those accepted by a judge, consistently outperforms the mask-only baseline both at the end of one epoch and at comparable update counts. The advantage persists when the dataset is larger, indicating that better coverage of the student's divergence states contributes to the gain.
Generality across harnesses
The framework is not tied to a single execution mechanism. The authors integrate R2S with a second harness, CaP-X, where the planner generates executable control code rather than invoking VLA skills. In this setting, SFT improves the 9B planner's success rate from 2.7% to 18.7%, and RL further raises it to 28.5%, exceeding the 27B teacher's 20.3%. Improvements are concentrated in the Object suites, where RL raises success rates from 39.0–49.0% to 67.0–72.0%. Performance remains low on Goal and Spatial, where the teacher rarely succeeds and provides little SFT data, but the authors note that the low absolute numbers reflect the difficulty of this harness rather than a failure of post-training.
Limitations
The findings are descriptive and limited by the evaluated cohort sizes. Three environments, two teachers, and four students were tested; the panel was not designed for multiplicity-adjusted confirmatory inference. Future work should test repeated samples, a broader model roster, and larger and more diverse task distributions to assess how well the findings generalize. The reward-free repair step assumes access to a teacher that can generate takeover sequences from any state, which may not hold for teacher architectures with different constraints.
What it means in practice
For practitioners building agentic systems, the key takeaway is that next-token scoring alone is insufficient for on-policy distillation when the student explores beyond the teacher's reliable region. R2S provides a concrete, reward-free method to repair the scorer using only teacher takeovers and consensus checks, without needing environment rewards during training. The repaired teacher can then be deployed as the standard scorer for OPD, enabling improved student performance without additional engineering overhead. The authors release the framework and checkpoints to support follow-up work.