RoFormer / RoPE

RoFormer: Enhanced Transformer with Rotary Position Embedding

Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu · arXiv 2021

Read paper · arXiv:2104.09864

Rotate adjacent projected feature pairs to express relative positions in attention logits.

What you will implement

Implement full-width RoPE; verify direction, frequencies, gradients and absolute offsets in chunked caches.

• 63. Rotary Position Embedding

  Rotate adjacent feature pairs and understand relative-position logits and cached-decoding offsets.

  Implement apply_rope; supplied experiments cover relative positions and chunked KV caching without RoFormer training or context-extension methods.