EWC

Overcoming catastrophic forgetting in neural networks

James Kirkpatrick et al. · PNAS 2017

Read paper · arXiv:1612.00796v2

Use a Fisher-diagonal parameter anchor to study the tradeoff between new-task learning and old-task retention.

What you will implement

Given old parameters and Fisher weights, implement the EWC penalty and its gradients, then observe the tradeoff in synthetic two-task training.

• 95. EWC: Penalize Changes to Important Parameters

  Implement EWC with a supplied Fisher diagonal and freeze the old-parameter and importance snapshots.

  One scalar penalty with a supplied Fisher diagonal; a two-task linear training experiment is provided. No Fisher estimation or LLM training.