Intermediate · 09
Seeing and hearing: multimodal models
How one model can read a photo, listen to speech and answer in words: everything becomes tokens.
6 min read · Tap underlined words for a definition
Swipe up to begin
A multimodal model works with more than one kind of data: text, images, audio, sometimes video. The trick that makes it possible is simple: turn everything into the same currency, a sequence of vectors the can attend over.
Images become patches
A vision transformer cuts a picture into a grid of small squares, say 16 × 16 pixels each. Each patch is flattened and turned into an embedding, exactly like a word.
To the transformer, a photo is just a sentence of a few hundred "patch words".
Sound becomes pictures, then tokens
Speech models like Whisper turn audio into a spectrogram, a picture of which frequencies are loud over time, then process it much like an image.
How does a vision transformer read an image?
Teaching images and words to meet
CLIP, a 2021 model, was trained on hundreds of millions of image-caption pairs. It learned to put a photo of a dog and the words "a photo of a dog" close together in the same space.
That shared space lets you search photos with words, and lets text models "see".
What they're good and bad at
Good: describing scenes, reading charts and handwriting, answering questions about screenshots.
Still shaky: counting many small objects, exact spatial reasoning ("is the cup left of the plate?"), and reading tiny text.
What did CLIP learn from image-caption pairs?
Before you drift off
- Multimodal models turn images and audio into token-like vectors.
- Vision transformers read images as grids of patches.
- Shared embedding spaces like CLIP's link pictures and words.
Terms in this lesson
Go deeper, free and open
- An Image is Worth 16x16 Words (ViT) · Dosovitskiy et al.
- Learning Transferable Visual Models From Natural Language Supervision (CLIP) · Radford et al.
- Hugging Face Computer Vision Course · Hugging Face community · Free
That's the lesson
Seeing and hearing: multimodal models · 6 min
Spotted a mistake? Suggest an edit
Previous: How we test language models