Multimodal reinforcement learning with verifiable rewards (RLVR) provides reliable task-level supervision, yet offers limited guidance for localizing errors in visual reasoning. On-policy self-distillation (OPSD) complements this supervision with dense token-level corrections from a privileged teacher equipped with an evidence-centered visual view. However, the resulting corrections may conflate visual evidence with non-visual shifts induced by linguistic priors, response style, or the privileged view itself. We formalize this phenomenon as privileged-correction contamination: a token-level correction is contaminated when its update direction cannot be consistently attributed to the privileged visual evidence. Directly distilling contaminated corrections may therefore reinforce erroneous token-level updates. To address this issue, we propose reinforcement learning with visually attributable on-policy self-distillation, an evidence-consistent paradigm that augments task-level RLVR supervision with privileged token-level corrections grounded in visual evidence. We instantiate this paradigm with Visual-Attribution Gating (VAG), which uses the verifier-derived advantage as the optimization anchor and validates each privileged correction against the contrast between evidence-present and evidence-degraded teacher views. By admitting only corrections directionally supported by this visual contrast, VAG preserves visually grounded guidance while suppressing inconsistent residuals. Experiments across six visual reasoning benchmarks show that VAG consistently outperforms GRPO and OPSD methods, establishing new state-of-the-art average performance.
From task-level RLVR to token-level localization
Multimodal reinforcement learning with verifiable rewards (RLVR) has become a standard framework for training models on tasks with verifiable ground truth, such as mathematical problem solving or execution-style reasoning. The verifier supplies a binary or scalar signal that indicates whether the model's output is correct. While this provides reliable task-level supervision, the reward signal is sparse: the model receives feedback only after producing a complete answer, making it difficult to pinpoint which intermediate tokens contributed to an error. On-policy self-distillation (OPSD) was introduced to fill this gap by adding dense token-level corrections. In OPSD, a privileged teacher—equipped with an evidence-centered visual view or additional annotations—produces token-level targets that the student model mimics. The intuition is that these dense targets guide the student toward more grounded intermediate representations, improving the efficiency of later policy updates.
However, the privilege carried by the teacher introduces a subtle but persistent problem. The teacher's view may include information that is not available to the student at inference time, such as ground-truth region masks, extra linguistic context, or a different visual perspective. When the student mimics these token-level targets, the resulting updates may reinforce patterns that depend on the privileged view rather than on what the student can actually observe. We formalize this as privileged-correction contamination: a token-level correction is contaminated when its update direction cannot be consistently attributed to the privileged visual evidence. Directly distilling such corrections risks reinforcing erroneous or spurious token-level associations.
Visual-Attribution Gating as the solution
To prevent contaminated corrections from influencing the student, the proposed paradigm—reinforcement learning with visually attributable on-policy self-distillation—introduces a gating mechanism that validates each privileged correction against the contrast between evidence-present and evidence-degraded teacher views. The core idea is simple but effective: for each token position, the system compares the teacher's output when the full visual evidence is available versus when that evidence is degraded or removed. If the correction's update direction aligns with the direction predicted by this visual contrast, the correction is admitted; otherwise, it is suppressed.
Formally, let A be the verifier-derived advantage signal from RLVR, and let C be the privileged token-level correction from the teacher. Visual-Attribution Gating (VAG) computes the directional alignment between C and the visual contrast ΔV = teacher(full evidence) − teacher(degraded evidence). Only when the inner product ⟨C, ΔV⟩ is positive is C used to update the student's policy. This ensures that every admitted correction is directionally supported by the presence of visual evidence, while inconsistent residuals are discarded.
Experimental results across six benchmarks
The authors evaluate VAG on six visual reasoning benchmarks that require grounded perception, numerical measurement, and multi-step inference. The comparison baselines include GRPO (a recent on-policy RL method) and OPSD (the vanilla on-policy self-distillation approach). Across all six benchmarks, VAG achieves higher average performance than both baselines, often by substantial margins. The results consistently show that privileging visually attributable corrections improves the model's ability to localize errors and refine its reasoning over time. The authors report new state-of-the-art average performance across the suite, with gains ranging from approximately 2 to 5 points over the strongest baseline depending on the benchmark.
What this means for practitioners
For engineers working with multimodal models on reasoning-intensive tasks, the VAG framework offers a practical method for improving model performance without requiring additional human-annotated data. The key insight—that token-level corrections should be validated against a visual contrast rather than accepted wholesale—is straightforward to implement on top of existing RLVR or self-distillation pipelines. The gating step can be computed using the same teacher model that already produces the corrections, adding minimal overhead.
The compute profile is also favorable: VAG does not require extra forward passes beyond what the teacher already generates for OPSD; it only adds a comparison step between the full and degraded views. This makes the approach attractive for deployment in resource-constrained settings where every forward pass counts.
Limitations and open directions
The authors acknowledge that the visual contrast used for gating relies on the ability to degrade or mask evidence in a controlled way. In some multimodal setups, constructing a meaningful "evidence-degraded" view may require careful engineering of the data pipeline. Additionally, the method's effectiveness depends on the quality of the privileged teacher; if the teacher's own representations are biased or narrow, the contrast may not surface the right corrections. The current work focuses on visual reasoning benchmarks; extending the framework to other multimodal domains such as video reasoning or cross-modal retrieval remains future work.
Nevertheless, the central contribution—formalizing privileged-correction contamination and providing a gating mechanism that filters contaminated updates—offers a reusable pattern for anyone building on-policy self-distillation or RLVR pipelines for multimodal models.