Beginner · 14
How AI makes pictures
Type a sentence, get an image. A gentle, no-maths look at how image generators turn noise into pictures, and where they fall short.
6 min read · Tap underlined words for a definition
Swipe up to begin
You type "a fox reading a book under a lamp, watercolour style" and a few seconds later there it is. It can feel like magic. The idea behind it is surprisingly gentle: start with static and tidy it up, one small step at a time.
Learning from captioned pictures
Image generators are trained on a very large number of images, each paired with a short description. From these pairs the model learns how words connect to shapes, colours, textures and styles.
It doesn't store the pictures like a photo album. It learns general patterns, such as what "fox" or "watercolour" tends to look like.
Starting from noise
Most modern image generators are diffusion models. During training, the model sees pictures with random speckles of noise added, a little at a time, until they are pure static. Its job is to learn to undo that: to guess, at each step, what a slightly cleaner picture would look like.
To make a new image, it begins with pure random noise and cleans it up step by step, guided by your words, until a picture appears.
How does a diffusion model make a new image?
Where they still stumble
Image generators can struggle with exact counts, readable text, hands and complicated scenes with many objects in specific places. They can also repeat biases from their training images, such as who they picture as a doctor or a nurse.
The copyright question
Many training images were collected from the internet, often without asking the artists. Whether that is allowed is being argued in courts and parliaments in several countries, and the answers are not settled yet.
There are also open questions about who owns an AI-made picture. If you use these tools for work, check the tool's terms and your local rules.
Which of these is a common weakness of AI image generators?
Before you drift off
- Image generators learn how words relate to pictures from huge sets of captioned images.
- Diffusion models start from noise and clean it up step by step, guided by your prompt.
- They still slip on fine details and raise unsettled questions about bias and copyright.
Terms in this lesson
Go deeper, free and open
- Denoising Diffusion Probabilistic Models · Ho, Jain & Abbeel, 2020
- High-Resolution Image Synthesis with Latent Diffusion Models · Rombach et al., 2022 · The paper behind Stable Diffusion
- Diffusion model · Wikipedia
That's the lesson
How AI makes pictures · 6 min
Spotted a mistake? Suggest an edit
Previous: Will AI take my job?