AdamW

Decoupled Weight Decay Regularization

Ilya Loshchilov, Frank Hutter · ICLR 2019

Read paper · arXiv:1711.05101

Decouple weight decay from the adaptive gradient update.

What you will implement

Implement moment estimates, bias correction, and a decoupled weight-decay step.

• 23. Decoupled AdamW Update

  Implement one AdamW step with bias correction and decoupled weight decay.

  Implement only adamw_step; state initialization and the two-dimensional optimization experiment are provided.