RMSNorm

Root Mean Square Layer Normalization

Biao Zhang and Rico Sennrich · NeurIPS 2019

Read paper · arXiv:1910.07467

Normalize by root mean square without centering and explore partial feature statistics.

What you will implement

Implement RMSNorm and first-k pRMSNorm; check scale identities and cross-feature gradients.

• 61. RMSNorm and Partial Feature Statistics

  Implement RMSNorm without mean centering and pRMSNorm with a first-k-feature scale estimate.

  Complete two normalization functions with explicit k and epsilon inside the square root; prefix-bias and offset experiments are supplied.