Stop 02 · Math & Statistics for AITopics · 4/13

Calculus Basics (Derivatives, Gradients)

In short

A model learns by asking one question over and over: if I nudge this weight, does the error go up or down? The derivative answers it. That's all training really is.

Derivatives measure how a function's output changes as its input changes — the mechanism every model uses to learn by adjusting weights to reduce error.

The purple blob pedals a bicycle up a hill that gets steeper towards the top.

Think of it like this

Cycling up a hill

On flat road your legs barely work. As the slope gets steeper, every metre is harder. The derivative is that steepness at the exact spot your cycle is on. The gradient is the same idea on a hill with many directions, pointing the steepest way up.

Watch it happen

Slope of f(x) = x² at x = 2

h = 1

h = 0.1

h = 0.01

h = 0.001

1/5Slope means rise over run. Step from x = 2 to 2 + h and measure how much f changes.

The big ideas

What you need to know

Derivative = instant slope

It tells you how fast the output changes when the input moves a tiny bit, right at one point.

A few rules cover most cases

d/dx of xⁿ is n·xⁿ⁻¹. Constants vanish. Sums split into separate derivatives.

Gradient = all slopes together

With many weights, the gradient is a vector of slopes, one per weight. It points towards steepest increase.

Training walks downhill

Gradient descent moves every weight a small step opposite the gradient, so the loss drops.

See it in code

Walk through the code

python

1def f(x):2    return x ** 23 4def slope(f, x, h=1e-5):5    return (f(x + h) - f(x - h)) / (2 * h)6 7print(round(slope(f, 2), 4))    # 4.0, matches 2x at x = 28 9# gradient descent on f: find the x that makes x**2 smallest10x = 5.011for step in range(5):12    grad = 2 * x                # derivative of x**213    x = x - 0.1 * grad          # step downhill14    print(step, round(x, 3))    # 4.0, 3.2, 2.56, 2.048, 1.638

1/4Estimate the slope by stepping a tiny bit either side of x.

The numeric slope checks your maths. The loop is gradient descent in miniature: x keeps moving towards 0, where x² is smallest.

Save yourself the pain

Common mistakes

Oops

Moving in the direction of the gradient to reduce loss.

Do this instead

The gradient points uphill. Subtract it to go downhill.

Oops

Using a huge learning rate to train faster.

Do this instead

Too big a step jumps over the minimum and the loss bounces or explodes. Start small, like 0.01 or 0.001.

Your mission

Race to the bottom

~20 min

Quick check

The derivative of the loss with respect to a weight is negative. To reduce the loss, what should you do to that weight?

Go deeper

1Refresh derivative rules (power, product, quotient) with practice problems.
2Learn what a gradient is: the vector of partial derivatives pointing toward steepest increase.
3Connect derivatives to the intuition of 'which direction reduces the loss.'