# Seeing and hearing: multimodal models · AI in Bed

Intermediate level · 6 min · https://www.aiinbed.com/learn/intermediate/multimodal-models

*How one model can read a photo, listen to speech and answer in words: everything becomes tokens.*

A **multimodal** model works with more than one kind of data: text, images, audio, sometimes video. The trick that makes it possible is simple: turn everything into the same currency, a sequence of vectors the transformer can attend over.

### Images become patches

A vision transformer cuts a picture into a grid of small squares, say 16 × 16 pixels each. Each patch is flattened and turned into an **embedding**, exactly like a word.

To the transformer, a photo is just a sentence of a few hundred "patch words".

### Sound becomes pictures, then tokens

Speech models like Whisper turn audio into a spectrogram, a picture of which frequencies are loud over time, then process it much like an image.

**Quick check:** How does a vision transformer read an image?

A. It converts the image to text with OCR first
B. It splits the image into patches and turns each into a vector, like word tokens
C. It looks at one pixel at a time in order
D. It can't; images need a separate non-transformer model

Answer: B. Patches become embeddings, and the transformer attends over them just as it attends over words.

### Teaching images and words to meet

CLIP, a 2021 model, was trained on hundreds of millions of image-caption pairs. It learned to put a photo of a dog and the words "a photo of a dog" close together in the same embedding space.

That shared space lets you search photos with words, and lets text models "see".

### What they're good and bad at

Good: describing scenes, reading charts and handwriting, answering questions about screenshots.

Still shaky: counting many small objects, exact spatial reasoning ("is the cup left of the plate?"), and reading tiny text.

**Quick check:** What did CLIP learn from image-caption pairs?

A. To generate videos
B. A shared space where matching images and texts are close together
C. To translate between languages
D. To remove noise from photos

Answer: B. Contrastive training pulls matching image-text pairs together and pushes mismatched ones apart.

**Key takeaways**

- Multimodal models turn images and audio into token-like vectors.
- Vision transformers read images as grids of patches.
- Shared embedding spaces like CLIP's link pictures and words.

**Go deeper (free sources)**

- [An Image is Worth 16x16 Words (ViT)](https://arxiv.org/abs/2010.11929) by Dosovitskiy et al.
- [Learning Transferable Visual Models From Natural Language Supervision (CLIP)](https://arxiv.org/abs/2103.00020) by Radford et al.
- [Hugging Face Computer Vision Course](https://huggingface.co/learn/computer-vision-course) by Hugging Face community