On-policy self-distillation has become a practical teacher-free approach for training language models, yet its privacy implications have received little attention. This paper fills that gap by conducting the first systematic study of membership inference attacks against on-policy self-distillation. The authors propose OP-Print, a unified attack framework that exploits the sparse, structured nature of the supervision imprint—concentrating on uncertainty-sensitive correction sites rather than the full rollout. OP-Print operates under both gray-box access (aggregating teacher-student residual corrections along the rollout) and black-box access (measuring restart-based output-basin stability as a surrogate for the hidden correction process). A key finding is that baseline attacks remain essentially at random guessing even under gray-box access, while OP-Print is broadly effective across diverse models and distillation schemes in both threat models. This demonstrates that on-policy self-distillation introduces genuine membership privacy risks that prior work had not identified, making it a significant contribution to ML security and privacy.
On-policy distillation removes the need for an external teacher by using the student's own outputs (enhanced with extra context or demonstrations) as supervision. This teacher-free design makes it practical for real-world training pipelines, but precisely because it's teacher-free and on-policy, the imprint of training examples on the final model may be particularly pronounced. The paper's motivation—that using two coupled views of the same training example may strengthen memorization—is intuitive and the resulting attack framework is both novel and practical.
The finding that all baseline attacks fail even under gray-box access, while OP-Print succeeds in both gray-box and black-box settings, is the paper's core contribution. It suggests that the privacy risk is real and that existing defense-oriented attacks are insufficient for this training paradigm. The sparsity and structure of the supervision imprint (concentrating on uncertainty-sensitive correction sites) is an insightful observation that explains why the attack works and may inform future defenses.
Given that on-policy self-distillation is becoming increasingly adopted as a practical approach (teacher-free, no need for separate teacher models, can leverage the model's own outputs), understanding its privacy implications is important for any organization deploying such systems. The paper thus addresses a timely and growing concern in machine learning practice.
While not a "capability jump" in the sense of improving model performance, it represents a meaningful advance in understanding the security and privacy trade-offs of a widely applicable training technique. The combination of novel attack methodology, analysis across threat models, and demonstration of risks in a practically relevant training paradigm makes this significant enough for tech news coverage.
Read the paper on arXiv