Advanced · 04

Fine-tuning on a budget: LoRA & quantisation

LoRA (low-rank adaptation) fine-tunes a large model by freezing its weights and training a few small extra matrices instead, so it needs a fraction of the memory. Quantisation stores the weights in fewer bits, such as 4-bit, and together (QLoRA) they let you fine-tune a billion-parameter model on one GPU.

6 min read · Tap underlined words for a definition

Fully a 70-billion-parameter model means updating 70 billion numbers and storing optimiser state for each, which needs a cluster. Parameter-efficient fine-tuning (PEFT) tricks let you get most of the benefit on a single GPU.

W🧊 frozend × d~16.8M numbers+B×Arank r = 8 · 🔥 trainable~65k numbers (0.4%)for one 4096 × 4096 layer
LoRA freezes the big weight matrix W and learns a small low-rank update B·A beside it.

LoRA: low-rank adaptation

Keep the original weights W frozen. Learn two thin matrices B (d×r) and A (r×d) with a small rank r like 8 or 16.

New output = W·x + B·A·x. For a 4096×4096 layer, that's ~65k trainable numbers at r=8 instead of ~16.8 million.

Swappable adapters

adapters are tiny files (megabytes). One base model can host many adapters: one for legal writing, one for your brand voice, one for SQL.

After training, B·A can be merged into W so costs nothing extra.

Quick check

In LoRA, which weights are updated during fine-tuning?

Quantisation

Store in fewer bits: 16-bit → 8-bit → 4-bit. A 4-bit model needs roughly ¼ the memory of 16-bit.

Clever schemes (GPTQ, AWQ, NF4) keep quality surprisingly close to the original.

QLoRA = both

Quantise the frozen base to 4-bit, then train LoRA adapters on top in higher .

This made fine-tuning a 65B model possible on a single 48 GB GPU.

Should you fine-tune at all?

Try good prompting and first. Fine-tune when you need a consistent style or format, a narrow skill, or a smaller cheaper model that matches a big one on your task.

Fine-tuning is great for behaviour; RAG is usually better for facts.

Quick check

Roughly how much memory does a 4-bit model need compared with the 16-bit version?

Before you drift off

  • LoRA trains tiny low-rank adapters while freezing the base.
  • Quantisation shrinks memory by storing weights in fewer bits.
  • QLoRA combines both for single-GPU fine-tuning.
  • Fine-tune for behaviour, use RAG for knowledge.

Terms in this lesson

Go deeper, free and open

Questions

What is LoRA in AI?

LoRA stands for low-rank adaptation. Instead of updating every weight in a model, it trains two small matrices per layer whose product is added to the frozen weights. The result is a small adapter file you can swap in and out.

What is quantisation in machine learning?

Quantisation (or quantization) stores a model's numbers with fewer bits, for example 4 or 8 instead of 16. It cuts memory and speeds up inference, with a small loss in quality.

What is QLoRA?

QLoRA combines the two: the frozen base model is quantised to 4-bit, and LoRA adapters are trained on top. It is the usual way to fine-tune a large open model on a single consumer GPU.

That's the lesson

Fine-tuning on a budget: LoRA & quantisation · 6 min

Each lesson you finish lights a star.

Finish & continue · Up nextDiffusion: art from noiseAdvanced · 6 min

Spotted a mistake? Suggest an edit

Previous: RAG & AI agents