Definition
Quantized low-rank adaptation keeps a pretrained base model frozen in a low-precision representation and backpropagates through it into trainable LoRA adapters. The base weights participate in forward and backward computation but are not updated. This cuts the memory required for the base while limiting optimizer state and gradients to the adapter parameters.
The QLoRA method
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer introduced QLoRA in 2023. Their method used a frozen 4-bit base, a 4-bit data type named NormalFloat, double quantization of the quantization constants, and paged optimizers for memory spikes. Those are specific parts of the published QLoRA system. "Quantized LoRA" is sometimes used more loosely for other low-bit bases and adapter setups.
QLoRA quantizes the base model, not necessarily every value in the training run or the final adapter. Computation and adapter weights may use higher precision. After training, deployment may keep the adapter separate, merge it into a dequantized base, or requantize a merged model. Each choice has different storage, latency, and compatibility consequences.
Lower memory use does not prove that the adapted model matches full fine-tuning. Results depend on the quantizer, target layers, rank, data, objective, base model, and evaluation. The adapter also remains tied to the exact compatible base and tokenizer.
Distinguish it from nearby terms
LoRA reduces the number of trainable parameters but can still load the frozen base at high precision. QLoRA adds base-model quantization to reduce that memory. Ordinary post-training quantization changes a trained model for deployment and does not itself train adapters.
Check your understanding
A QLoRA adapter works with the original base checkpoint but fails after the team swaps in a newer quantized revision. Which base, tokenizer, target-module, rank, and quantization metadata should have been versioned with the adapter?