Transformer

Attention Is All You Need

Ashish Vaswani et al. · NeurIPS 2017

Read paper · arXiv:1706.03762

Combine sequence information through multi-head attention, positional encoding, position-wise feed-forward networks, and residual normalization.

What you will implement

Independently implement scaled dot-product attention, multi-head attention, LayerNorm and Add & Norm, sinusoidal positional encoding, and position-wise feed-forward networks; helpers and small experiments are supplied.

• 22. Scaled Dot-Product Attention

  Implement scaled dot-product attention and visibility masking with a multi-head tensor layout.

  Implement only scaled_dot_product_attention; causal_mask and the prefix-average experiment are provided.

• 27. Multi-Head Attention

  Project inputs into separate head subspaces, attend independently, concatenate, and output-project.

  Implement only multi_head_attention; the attention kernel and two-head routing experiment are supplied.

• 28. LayerNorm and Add & Norm

  Implement token-local LayerNorm and the original Transformer's Add & Norm.

  Implement layer_norm and post_norm_residual; sublayer outputs and the training-free toy are supplied, with dropout disabled.

• 29. Sinusoidal Positional Encoding

  Implement paired sine/cosine position features at geometrically spaced frequencies.

  Implement only sinusoidal_encoding; the relative-offset experiment is provided.

• 30. Position-wise Feed-Forward Network

  Implement the shared two-layer ReLU network applied independently to each token.

  Implement only positionwise_ffn; training and data are provided.