Advanced · 04
Fine-tuning on a budget: LoRA & quantisation
LoRA (low-rank adaptation) fine-tunes a large model by freezing its weights and training a few small extra matrices instead, so it needs a fraction of the memory. Quantisation stores the weights in fewer bits, such as 4-bit, and together (QLoRA) they let you fine-tune a billion-parameter model on one GPU.
6 min read · Tap underlined words for a definition
Swipe up to begin
Fully a 70-billion-parameter model means updating 70 billion numbers and storing optimiser state for each, which needs a cluster. Parameter-efficient fine-tuning (PEFT) tricks let you get most of the benefit on a single GPU.
LoRA: low-rank adaptation
Keep the original weights W frozen. Learn two thin matrices B (d×r) and A (r×d) with a small rank r like 8 or 16.
New output = W·x + B·A·x. For a 4096×4096 layer, that's ~65k trainable numbers at r=8 instead of ~16.8 million.
Swappable adapters
adapters are tiny files (megabytes). One base model can host many adapters: one for legal writing, one for your brand voice, one for SQL.
After training, B·A can be merged into W so costs nothing extra.
In LoRA, which weights are updated during fine-tuning?
Quantisation
Store in fewer bits: 16-bit → 8-bit → 4-bit. A 4-bit model needs roughly ¼ the memory of 16-bit.
Clever schemes (GPTQ, AWQ, NF4) keep quality surprisingly close to the original.
QLoRA = both
Quantise the frozen base to 4-bit, then train LoRA adapters on top in higher .
This made fine-tuning a 65B model possible on a single 48 GB GPU.
Should you fine-tune at all?
Try good prompting and first. Fine-tune when you need a consistent style or format, a narrow skill, or a smaller cheaper model that matches a big one on your task.
Fine-tuning is great for behaviour; RAG is usually better for facts.
Roughly how much memory does a 4-bit model need compared with the 16-bit version?
Before you drift off
- LoRA trains tiny low-rank adapters while freezing the base.
- Quantisation shrinks memory by storing weights in fewer bits.
- QLoRA combines both for single-GPU fine-tuning.
- Fine-tune for behaviour, use RAG for knowledge.
Terms in this lesson
Go deeper, free and open
- LoRA: Low-Rank Adaptation of Large Language Models · Hu et al., 2021
- QLoRA: Efficient Finetuning of Quantized LLMs · Dettmers et al., 2023
- PEFT library documentation · Hugging Face · Apache 2.0
Questions
What is LoRA in AI?
LoRA stands for low-rank adaptation. Instead of updating every weight in a model, it trains two small matrices per layer whose product is added to the frozen weights. The result is a small adapter file you can swap in and out.
What is quantisation in machine learning?
Quantisation (or quantization) stores a model's numbers with fewer bits, for example 4 or 8 instead of 16. It cuts memory and speeds up inference, with a small loss in quality.
What is QLoRA?
QLoRA combines the two: the frozen base model is quantised to 4-bit, and LoRA adapters are trained on top. It is the usual way to fine-tune a large open model on a single consumer GPU.
That's the lesson
Fine-tuning on a budget: LoRA & quantisation · 6 min
Each lesson you finish lights a star.
Spotted a mistake? Suggest an edit
Previous: RAG & AI agents