LLM Frontier
Practice LLM Frontier in Python with research-based exercises, implementation guides, and tests on PaperLeet.
- 21. Transformer
Combine sequence information through multi-head attention, positional encoding, position-wise feed-forward networks, and residual normalization.
NeurIPS 2017 · 5 exercises - 47. RMSNorm
Normalize by root mean square without centering and explore partial feature statistics.
NeurIPS 2019 · 1 exercises - 49. RoFormer / RoPE
Rotate adjacent projected feature pairs to express relative positions in attention logits.
arXiv 2021 · 1 exercises - 48. GQA
Share KV heads across contiguous query groups and convert multi-head projections by group-wise averaging.
EMNLP 2023 · 1 exercises - 22. AdamW
Decouple weight decay from the adaptive gradient update.
ICLR 2019 · 1 exercises - 23. FlashAttention
Accumulate exact attention in blocks with online softmax without materializing the full attention matrix.
NeurIPS 2022 · 1 exercises - 13. LoRA
Freeze pretrained weights and learn an update through a low-rank matrix product.
ICLR 2022 · 1 exercises - 26. Kimi K3
Implement Kimi K3's core mechanisms for sequence mixing, depth aggregation, and bounded activations: KDA, Block AttnRes, and SiTU-GLU.
arXiv 2026 · 3 exercises - 25. GLM-5
Extract head-wise orthogonalized updates and asynchronous-RL importance corrections from the GLM-5 technical report.
arXiv 2026 · 2 exercises