SDPO

Reinforcement Learning via Self-Distillation

Jonas Hübotter et al. · ICML 2026

Read paper · arXiv:2601.20802

Turn environment feedback and successful attempts into token-level self-distillation.

What you will implement

Same-question feedback routing, reverse KL, and top-k tail-mass coarsening.

• 38. SDPO Feedback Selection and Self-Distillation

  Select a successful self/peer solution or environment feedback following Table 2, then distill.

  Implement feedback routing and token-masked reverse KL; tiny programs and a categorical teacher are supplied. Choose the lowest index among multiple peer solutions.

• 39. SDPO Top-K and Tail-Mass KL

  Coarsen the vocabulary into student top-k entries and one tail bin while preserving total probability mass.

  Implement top-k selection, complementary logsumexp, and reverse KL; tests and information curves are supplied.