Progressive Distillation

Progressive Distillation for Fast Sampling of Diffusion Models

Tim Salimans, Jonathan Ho · ICLR 2022

Read paper · arXiv:2202.00512

Distill two deterministic teacher steps into one student step, progressively reducing the sampling budget.

What you will implement

Implement v-parameterization conversions, then derive and validate the two-teacher-step to one-student-step distillation target.

• 7. v-Parameterization and Inverse Rotation

  Implement Appendix D’s v-parameterization and division-free inverse rotation to understand endpoint stability.

  Selected parameterization module: implement make_v, v_to_x0, and v_to_eps. Teacher–student distillation training is outside this module.

• 26. Progressive Distillation: Two Teacher Steps to One

  Derive the exact clean prediction that makes one student DDIM step match two teacher steps, then learn it.

  Implement ddim_step, distilled_target, distillation_loss. toy.py provides the training and visualization scaffold.