QLoRA

QLoRA: Efficient Finetuning of Quantized LLMs

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer · NeurIPS 2023

Read paper · arXiv:2305.14314v1

Train low-rank adapters on a quantized frozen base to reduce fine-tuning memory.

What you will implement

Implement blockwise NF4 quantization, dequantization, and adapter forward; double quantization and paged optimizers are outside this exercise.

• 15. QLoRA NF4 Blocks and Trainable Adapters

  Implement block NF4 quantization and train adapters through a frozen quantized base.

  Three CPU functions. Code indices are unpacked uint8 for inspectability; packed 4-bit storage, double quantization and paged optimizers are outside scope.