Differential Transformer

Differential Transformer

Tianzhu Ye et al. · ICLR 2025 Oral

Read paper · arXiv:2410.05258

Subtract two attention maps to suppress common-mode noise.

What you will implement

Differential attention on projected tensors and lambda parameterization.

• 42. Differential Transformer Attention

  Subtract two attention maps to cancel shared responses and propagate gradients through a learned lambda.

  Implement projected differential attention and lambda; per-head normalization, output projection and the full language model are outside this module.