Stop 02 · Math & Statistics for AITopics · 5/13

Chain Rule

In short

A neural network is functions inside functions inside functions. The chain rule is how you find the effect of a weight deep inside on the final error. It's the maths behind backpropagation.

The calculus rule for differentiating composed functions — the exact mathematical mechanism behind backpropagation in neural networks.

The purple blob turns a crank connected to three meshing gears of different sizes.

Think of it like this

Gears in a row

Turn a big gear once and the middle gear turns 3 times. The middle one turns the small gear 2 times for each of its turns. So the small gear spins 3 × 2 = 6 times. The chain rule works the same way: multiply the rates along the chain.

Watch it happen

y = (3x + 1)² at x = 2

x = 2

u = 3x + 1

y = u²

dy/dx

1/4Forward pass starts with the input x = 2.

The big ideas

What you need to know

Multiply along the chain

If y depends on u and u depends on x, then dy/dx = dy/du × du/dx.

Each layer is one link

In a network, each layer only needs its own local derivative. The chain rule stitches them together.

Backprop runs it backwards

Start at the loss and multiply local derivatives layer by layer towards the input. That reuses work and is fast.

Frameworks do it for you

PyTorch records every operation and applies the chain rule when you call .backward().

See it in code

Walk through the code

python

1# by hand2x = 2.03u = 3 * x + 1          # inner: u = 74y = u ** 2             # outer: y = 495dy_du = 2 * u          # 146du_dx = 37print(dy_du * du_dx)   # 42.08 9# the same with PyTorch autograd10import torch11xt = torch.tensor(2.0, requires_grad=True)12yt = (3 * xt + 1) ** 213yt.backward()14print(xt.grad)         # tensor(42.)

1/4Forward pass: compute the inner value, then the outer value.

Both give 42. PyTorch builds the chain of operations as you compute, then backward() multiplies the local derivatives for you.

Save yourself the pain

Common mistakes

Oops

Adding the local derivatives instead of multiplying them.

Do this instead

Along one chain you multiply. You add only when a variable affects the output through two separate paths.

Oops

Forgetting that the outer derivative uses the inner value, like writing 2x instead of 2u.

Do this instead

Evaluate each local derivative at the value that flowed into it during the forward pass.

Your mission

Backprop by hand

~20 min

Quick check

u = 5x and y = u². At x = 1, what is dy/dx?

Go deeper

1Practice the chain rule on nested functions by hand.
2Trace a tiny 2-layer neural network's backward pass and identify each chain-rule step.
3Understand why backpropagation is 'just' repeated application of the chain rule.