Stop 10 · Computer VisionTopics · 6/9

Vision Transformers (ViT)

In short

Transformers took over text, and then someone asked: what if an image were just a sentence of little squares? That idea, ViT, now rivals or beats CNNs on many vision tasks.

Applying the Transformer architecture to images by splitting them into patches treated like tokens — now competitive with or better than CNNs on many vision benchmarks.

The blob lays out nine square tiles of a tiger picture in one neat row on a table.

Think of it like this

A photo cut into cards

Cut a tiger photo into nine square cards and lay them in a row. Each card alone is just stripes or an eye. But if every card can look at every other card, together they clearly say 'tiger'. ViT treats each patch like a word in a sentence.

Watch it happen

From pixels to a class with ViT

Image

Patches

Embed + pos

Encoder

Class

1/5A 224x224 image goes in.

The big ideas

What you need to know

Patches are tokens

ViT doesn't use convolutions to scan the image. It cuts it into patches and treats them like words.

Global view from layer one

Attention links far-apart patches immediately. CNNs only see far areas after many layers stack up.

Data hungry

With little data, ViTs tend to lag CNNs. Pretrained on huge datasets, they transfer very well.

Position must be added

Attention has no sense of order, so position embeddings tell the model where each patch sits.

See it in code

Walk through the code

python

1from transformers import pipeline2 3clf = pipeline("image-classification",4               model="google/vit-base-patch16-224")5 6for pred in clf("tiger.jpg", top_k=3):7    print(pred["label"], round(pred["score"], 3))8 9# Patch maths for this model:10# 224 / 16 = 14 patches per side -> 196 patches + 1 [CLS] token

1/3The model name tells you the patch size (16) and the input size (224).

A ViT-Base model with 16x16 patches, pretrained on ImageNet-21k and fine-tuned on ImageNet-1k.

Save yourself the pain

Common mistakes

Oops

Training a ViT from scratch on 2,000 images.

Do this instead

Start from a pretrained ViT and fine-tune. From scratch, a small CNN will likely do better on small data.

Oops

Assuming a bigger input image costs the same.

Do this instead

Doubling width and height gives four times the patches, and attention cost grows with the square of token count.

Your mission

ViT vs ResNet on your photos

~25 min

Quick check

A ViT uses 16x16 patches on a 224x224 image. How many patch tokens does it create?

Go deeper

1Read how ViT splits an image into patches and embeds them like sequence tokens.
2Fine-tune a pretrained ViT on a small image classification dataset.
3Compare ViT and CNN performance/data requirements at a conceptual level.