Stop 02 · Math & Statistics for AITopics · 13/13

Optimization Basics (Gradient Descent)

In short

Almost every model, from a simple regression to a giant neural network, learns the same way: take small steps that make the error a bit smaller. That's gradient descent.

The iterative algorithm that adjusts model parameters in the direction that reduces loss the fastest — the core training loop of virtually every ML and DL model.

The mascot walks down a dark valley at night, pointing a torch at its feet.

Think of it like this

Walking down a dark valley

It's night and your torch only lights the ground at your feet. To reach the bottom, you check which way slopes down, take a step, and check again. Steep ground, bigger steps. Nearly flat, you're almost there.

Watch it happen

w after each step (loss = (w − 3)², lr = 0.1)

0.6

1.08

1.46

1.77

…

1/5Start at w = 0. The slope 2(w − 3) is −6, so moving right lowers the loss.

The big ideas

What you need to know

Step against the gradient

New weight = old weight − learning rate × gradient. That moves you downhill on the error.

Learning rate sets step size

Too small and training crawls. Too big and you overshoot and the loss can blow up.

Mini-batches save time

Stochastic and mini-batch versions use a few rows per step instead of the whole dataset.

Not every valley is simple

Complex models have bumpy error surfaces. You may land in a good spot, not the perfect one.

See it in code

Walk through the code

python

1def loss(w):2    return (w - 3) ** 23 4def grad(w):5    return 2 * (w - 3)6 7w, lr = 0.0, 0.18for step in range(50):9    w = w - lr * grad(w)10print(round(w, 4), round(loss(w), 6))   # 3.0 0.0

1/4The loss we want to shrink and its slope (gradient).

Gradient descent in four lines. Real libraries do the same thing with millions of weights.

Save yourself the pain

Common mistakes

Oops

Setting a huge learning rate to 'train faster'.

Do this instead

Try lr = 1.1 in the code above. w jumps further each step. Start small and tune.

Oops

Running gradient descent on features with wildly different scales.

Do this instead

Scale features first. Otherwise the steps zigzag and training takes far longer.

Your mission

Break and fix the learning rate

~15 min

Quick check

Your training loss jumps up and down and keeps getting bigger. What's the most likely fix?

Go deeper

1Implement gradient descent from scratch on a simple 1D or 2D loss function.
2Visualize how learning rate affects convergence (too small, too large, just right).
3Learn the difference between batch, stochastic and mini-batch gradient descent.