OPSD

Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models

Siyan Zhao et al. · arXiv 2026

Read paper · arXiv:2601.18734

Use a privileged-context teacher from the same model to supervise student rollouts.

What you will implement

Dual-context causal alignment and pointwise divergence clipping.

• 36. OPSD Dual-Context Rollout Alignment

  Align next-token logits for the same student response under unequal context lengths.

  Implement causal position selection, variable-length masking, and teacher detachment; logits and a toy training loop are supplied.

• 37. OPSD Pointwise Divergence Clipping

  Upper-clip vocabulary contributions to forward and reverse KL.

  Implement signed divergence terms, pointwise clipping, and response-level averaging; validation and experiments are supplied.