Stop 10 · Computer VisionTopics · 5/9

Image Segmentation (Overview)

In short

A box around a tumour or a road is too rough when you need the exact shape. Segmentation labels every single pixel, so you know precisely where one thing ends and another begins.

Classifying every pixel in an image (semantic) or every pixel per object instance (instance segmentation) — finer-grained than detection, used in medical imaging and autonomous driving.

The blob fills every region of a chalk rangoli outline with a different coloured powder.

Think of it like this

Filling in the rangoli

Drawing a rangoli outline is easy. Filling it is the real work: every little region gets its own colour and no gap is left blank. Segmentation fills the image the same way, giving every pixel a class like road, sky or person.

Watch it happen

Two kinds of segmentation

Semantic

One class per pixel
All people share one colour
Good for roads, sky, land

Instance

Each object gets its own mask
Person 1 and person 2 differ
Good for counting things

Use semantic when you care about 'what kind of stuff is here'. Use instance when you need to separate and count individual objects.

The big ideas

What you need to know

Output is a mask

The model returns an array the same size as the image, with a class number in every pixel.

Encoder-decoder shape

U-Net style models shrink the image to understand it, then grow it back to full size to label each pixel.

Panoptic does both

Panoptic segmentation combines semantic labels for background stuff with instance masks for countable objects.

Measure with IoU

Compare predicted and true masks with IoU or Dice. Pixel accuracy misleads when background covers most of the image.

See it in code

Walk through the code

python

1import torch2from torchvision.models.segmentation import deeplabv3_resnet50, DeepLabV3_ResNet50_Weights3from PIL import Image4 5weights = DeepLabV3_ResNet50_Weights.DEFAULT6model = deeplabv3_resnet50(weights=weights).eval()7prep = weights.transforms()8 9img = Image.open("street.jpg").convert("RGB")10with torch.no_grad():11    out = model(prep(img).unsqueeze(0))["out"]   # (1, classes, H, W)12 13mask = out.argmax(1)[0]                          # (H, W) class id per pixel14print(mask.shape, mask.unique())

1/3Load pretrained weights and the matching preprocessing, so input is handled correctly.

A pretrained DeepLabV3 gives a class for every pixel. argmax picks the most likely class at each spot.

Save yourself the pain

Common mistakes

Oops

Using plain pixel accuracy when 90 percent of the image is background.

Do this instead

A model predicting 'background' everywhere scores 90 percent. Use per-class IoU or Dice instead.

Oops

Using segmentation when boxes would do.

Do this instead

Pixel labels are slow and costly to annotate. If you only need to count or locate, detection is cheaper.

Your mission

Colour a street scene

~30 min

Quick check

A photo has three people standing together. Semantic segmentation would:

Go deeper

1Run a pretrained segmentation model and visualize the pixel-level mask output.
2Learn the difference between semantic and instance segmentation.
3Understand common use cases where segmentation is required over detection.