Stop 07 · Deep Learning FoundationsTopics · 4/11

Forward & Backward Propagation

In short

Training a network is a loop of two passes. Forward makes a guess. Backward works out which weight deserves how much blame. Get this, and deep learning stops feeling like magic.

Forward propagation computes predictions layer by layer; backward propagation computes gradients layer by layer in reverse — together, the training loop of every neural network.

The blob passes a tiffin box along a chain of helpers, then a note travels back along the same chain.

Think of it like this

Tracing a wrong tiffin

A dabba reaches the wrong office. The complaint travels back down the chain: the delivery man, the station sorter, the collector. Each one checks how much their step caused the mix-up and fixes it. Backprop passes blame backwards the same way.

Watch it happen

One training step

Input x

Hidden h

Output ŷ

Loss

h = tanh(w1·x)

1/6Forward: the input is multiplied by w1 and squashed by tanh to make the hidden value.

The big ideas

What you need to know

Forward pass

Data flows layer by layer to a prediction. Intermediate values are saved because the backward pass needs them.

Backward pass

Starting from the loss, gradients flow back layer by layer using the chain rule. Each weight gets its own gradient.

Chain rule does the work

The gradient for an early weight is the product of local slopes along the path from that weight to the loss.

Autograd automates it

PyTorch records the forward pass and computes every gradient when you call loss.backward(). You still need to understand it.

See it in code

Walk through the code

python

1import numpy as np2 3x, y = 2.0, 1.0           # one input, one target4w1, w2 = 0.5, -0.3        # two weights in a chain5 6# forward pass7h = np.tanh(w1 * x)       # hidden activation8y_hat = w2 * h            # output9loss = (y_hat - y) ** 210 11# backward pass: chain rule, from the loss back to each weight12d_yhat = 2 * (y_hat - y)13d_w2 = d_yhat * h14d_h = d_yhat * w215d_w1 = d_h * (1 - h ** 2) * x   # tanh'(z) = 1 - tanh(z)^216print(loss, d_w1, d_w2)

1/3Forward: compute h, then the prediction, then the loss. Keep h, the backward pass needs it.

The same chain as the scene, done by hand. Each backward line multiplies by one local slope.

Save yourself the pain

Common mistakes

Oops

Thinking backprop is a separate learning algorithm from gradient descent.

Do this instead

Backprop only computes gradients. Gradient descent (SGD, Adam) then uses them to update the weights.

Oops

Forgetting that the backward pass needs values from the forward pass.

Do this instead

That's why training uses more memory than inference: activations are stored until gradients are computed.

Your mission

Check your gradients

~25 min

Quick check

During backprop, how is the gradient for a weight in an early layer found?

Go deeper

1Trace forward and backward passes by hand on a tiny 2-layer network.
2Implement a manual backward pass in NumPy before relying on autograd.
3Learn how PyTorch's autograd automates this process under the hood.