VICReg

VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning

Adrien Bardes, Jean Ponce, Yann LeCun · ICLR 2022

Read paper · arXiv:2105.04906

Learn representations with variance, view invariance, and decorrelation constraints.

What you will implement

Batch statistics, weights, and gradients of the three loss terms.

• 54. VICReg Variance, Invariance and Covariance

  Implement an explicit anti-collapse and redundancy objective, separating a variance floor from covariance penalties.

  Implement three scalar terms and their weighted sum on two feature views. Encoder, projector and distributed gathering are outside the module.