Stop 07 · Deep Learning FoundationsTopics · 6/11

Gradient Descent Variants (SGD, Adam)

In short

The gradient tells you which way is downhill. The optimiser decides how big a step to take and how to use past steps. That choice decides whether training takes minutes or never finishes.

Optimization algorithms that update model weights using gradients — SGD is the simplest, Adam adds momentum and adaptive learning rates and is the default for most deep learning.

The blob carefully walks down a misty green hill with a lantern, feeling the slope with each step.

Think of it like this

Walking down a foggy hill

You're on a hill in Munnar at night, thick fog, no view. You feel the slope under your feet and step downhill. SGD takes the same-size step each time. Adam is like smart shoes that take bigger strides on gentle slopes and careful ones on steep, bumpy ground.

Watch it happen

Two popular optimisers

SGD (+ momentum)

One learning rate for every weight
Momentum keeps rolling past bumps
Needs careful learning rate tuning

Adam / AdamW

Adapts step size per weight
Works well with default settings
Default for transformers and most projects

Start with AdamW at a learning rate around 1e-3 (lower for fine-tuning). Try SGD with momentum when you have time to tune, as it's still common for image models.

The big ideas

What you need to know

The update rule

new weight = old weight - learning rate × gradient. Every optimiser is a variation on this one line.

Mini-batches

'Stochastic' means each step uses a small random batch, not all the data. Noisier, but far cheaper per step.

Momentum

Momentum adds a fraction of the previous step, like a cycle rolling downhill. It speeds up through flat stretches.

Learning rate is king

Too high and the loss jumps around or explodes. Too low and training crawls. Schedules lower it as training goes on.

How it works

What Adam does on each step

1
Get the gradient
Backprop gives the gradient for each weight on this batch.
2
Update the running average
Keep a smoothed average of recent gradients, like momentum.
3
Track gradient size
Keep a smoothed average of squared gradients for each weight.
4
Take a scaled step
Divide the average by the square root of the size, so noisy weights take smaller steps.

See it in code

Walk through the code

python

1import torch2 3def train(opt_name, steps=100):4    w = torch.tensor([0.0], requires_grad=True)5    opt = (torch.optim.SGD([w], lr=0.1) if opt_name == "sgd"6           else torch.optim.Adam([w], lr=0.1))7    for _ in range(steps):8        opt.zero_grad()9        loss = (w - 3) ** 2      # lowest point is at w = 310        loss.backward()11        opt.step()12    return w.item()13 14print("SGD: ", train("sgd"))15print("Adam:", train("adam"))

1/3One weight starting at 0, and the chosen optimiser with the same learning rate.

Both optimisers find the bottom of a simple bowl. On real networks with millions of weights, their paths look very different.

Save yourself the pain

Common mistakes

Oops

Forgetting opt.zero_grad(), so gradients pile up across steps.

Do this instead

PyTorch adds new gradients to old ones. Clear them every step before calling backward().

Oops

Copying a learning rate from a tutorial with a different optimiser.

Do this instead

Good learning rates differ a lot between SGD and Adam. Run a quick sweep, like 1e-4, 1e-3, 1e-2.

Your mission

Optimiser race

~25 min

Quick check

Your training loss suddenly shoots up to a huge number after a few steps. What's the first thing to try?

Go deeper

1Train the same model with SGD and Adam and compare convergence speed.
2Learn what momentum and adaptive learning rates conceptually solve.
3Understand learning rate scheduling and why a fixed rate is often suboptimal.