Intermediate · 10

Open-weight models and running AI yourself

What "open" really means for AI models, why it matters, and how a laptop can run a capable chatbot.

6 min read · Tap underlined words for a definition

Some AI models are only available through a company's app or API. Others publish their weights, the learned numbers, so anyone can download and run them. These are called open-weight models.

Open weights isn't always open source

Truly open-source AI would also share the training code and enough detail about the data to rebuild the model. Many "open" models share only the , often under a licence with conditions.

Always read the licence before building a product on one.

Why people want open models

Privacy: data never leaves your machine. Control: the model won't change under you overnight. Customisation: you can fine-tune it on your own data. Research: scientists can look inside.

Quick check

What does an open-weight model let you do that an API-only model doesn't?

Fitting a model on a laptop

A model with 8 billion parameters stored as 16-bit numbers needs about 16 GB just for weights. Quantisation stores each weight in 4 bits instead, cutting that to about 4 to 5 GB with a small quality loss.

Tools like llama.cpp and Ollama do this for you, so many modern laptops can run a decent offline.

Run itHow much memory does a model need?

def memory_gb(billion_params, bits):
    return billion_params * 1e9 * bits / 8 / 1e9

for size in [1, 8, 70]:
    row = [f"{bits}-bit: {memory_gb(size, bits):5.1f} GB" for bits in (16, 8, 4)]
    print(f"{size:>3}B params  " + "   ".join(row))

Edit anything, then run. Python runs in your browser.

Weights only. Running the model needs some extra memory for the conversation, so add 10 to 20 percent.

Quick check

Roughly how much memory do the weights of an 8-billion-parameter model need at 4 bits?

Before you drift off

Open-weight models publish their learned numbers; open source goes further.
Open models give privacy, control and room to customise.
Quantisation shrinks models enough to run on ordinary laptops.

Terms in this lesson

Go deeper, free and open

The Open Source AI Definition · Open Source Initiative
llama.cpp · ggml-org · MIT
Ollama · Ollama · MIT

That's the lesson

Open-weight models and running AI yourself · 6 min

Finish & continue · Up nextDecision trees and random forestsIntermediate · 7 min

Was this lesson clear?

Spotted a mistake? Suggest an edit

Previous: Seeing and hearing: multimodal models