Stop 07 · Deep Learning FoundationsTopics · 10/11

Dropout & Regularization in DL

In short

Neural networks have so many weights that they can memorise a training set outright. Dropout and its friends make memorising harder, so the network has to learn patterns that hold up.

Randomly zeroing out neurons during training to prevent co-adaptation and overfitting — one of the simplest and most effective regularization techniques in deep learning.

The blob as a cricket coach waves some players onto the bench while the rest practise at the nets.

Think of it like this

Random players sit out practice

A cricket coach benches a few random players at each net session. Nobody can rely on the star batter, so everyone learns to bat and bowl. On match day, the full team plays. Dropout trains a network the same way.

Watch it happen

Dropout with p = 0.5 on six activations

0.8

0.2

0.5

0.9

0.4

0.7

1/4Six activations coming out of a hidden layer for one example.

The big ideas

What you need to know

Dropout zeroes random units

With probability p, each activation is set to 0 during training. PyTorch scales the rest by 1/(1-p).

Weight decay keeps weights small

Weight decay is the deep learning version of L2. AdamW applies it correctly, separate from the gradient step.

Early stopping

Watch validation loss and stop when it starts rising. Save the best checkpoint, not the last one.

Data augmentation

Flip, crop or slightly rotate training images. More varied data is often the strongest regulariser of all.

See it in code

Walk through the code

python

1import torch2import torch.nn as nn3 4model = nn.Sequential(5    nn.Linear(100, 256), nn.ReLU(),6    nn.Dropout(p=0.5),            # zero half the activations, randomly7    nn.Linear(256, 10),8)9opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.01)10 11x = torch.randn(4, 100)12model.train()13print(model(x)[0, :3])   # changes every call: different units dropped14model.eval()15print(model(x)[0, :3])   # same every call: dropout is switched off

1/4Dropout sits after the activation. p=0.5 is common for wide layers; smaller values suit smaller layers.

Dropout and weight decay together. Run the train-mode print twice and the numbers change; the eval-mode print stays fixed.

Save yourself the pain

Common mistakes

Oops

Leaving the model in train mode during evaluation.

Do this instead

Dropout keeps firing in train mode, so accuracy looks worse and changes every run. Call model.eval() first.

Oops

Cranking dropout to 0.8 because the model overfits.

Do this instead

Too much dropout makes the model underfit. Try 0.1 to 0.5, plus early stopping and more data.

Your mission

Tame an overfitting network

~35 min

Quick check

Why does PyTorch scale up the surviving activations during dropout training?

Go deeper

1Add dropout to a network and observe its effect on train vs. validation loss.
2Learn typical dropout rates for different layer types.
3Understand why dropout is disabled at inference time.