Researchers have proposed a new method for on-policy distillation called Preference-Based Self-Distillation (PBSD). Unlike previous methods that rely on KL matching to the teacher model, PBSD uses reward regularization to derive a target policy. This approach optimizes preference gaps between teacher and student samples while maintaining on-policy student sampling. Experiments show that PBSD achieves stronger average performance than comparable baselines, with improved training stability and token efficiency. --- Why it matters: This matters because it addresses limitations in existing self-distillation methods, which can suffer from training instability and degrade reasoning performance over time. By providing a reward-regularized perspective, PBSD offers a more robust approach to on-policy distillation. Source: https://arxiv.org/abs/2605.05040