DistiLLM-2

DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs

Jongwoo Ko et al. · ICML 2025 Oral

Read paper · arXiv:2503.07067v2

Use distinct distillation directions on teacher- and student-generated responses.

What you will implement

Skew KL on teacher trajectories and skew reverse KL on student trajectories.

• 44. DistiLLM-2 Two-source Asymmetric Distillation

  Implement teacher-response SKL, student-response SRKL and the paired Eq.2 loss.

  Fixed coefficients and full conditional vocabularies; responses are supplied, retaining the 1/(2B) reduction.