LoRA

LoRA: Low-Rank Adaptation of Large Language Models

Edward J. Hu et al. · ICLR 2022

Read paper · arXiv:2106.09685v2

Freeze pretrained weights and learn an update through a low-rank matrix product.

What you will implement

Implement the low-rank update and linear forward pass; verify frozen parameters and a small regression experiment.

• 14. LoRA Low-rank Updates and Frozen-base Forward

  Implement a scaled low-rank update while freezing base parameters.

  Two functions for a linear layer; initialization and a rank-two regression training loop are provided.