Neural Network from Scratch (NumPy)
Part of the AI Engineer Roadmap
In short
Libraries hide the maths behind one call to backward(). Build a small network with only NumPy once, and you'll never again wonder what's happening inside one.
Implementing forward pass, backward pass and gradient descent using only NumPy — the single best exercise for truly understanding what frameworks automate for you.
Think of it like this
Chai from scratch
Tea bags are convenient, but making chai properly means crushing the ginger, boiling the leaves, adding milk at the right time. Do it once and you understand why it tastes the way it does. Coding a network in NumPy is chai from scratch.
Watch it happen
Your build plan
1/6Create random weight matrices W1 (2 × 8) and W2 (8 × 1), and zero biases.
The big ideas
What you need to know
Everything is matrix maths
A whole batch goes through a layer as one matrix multiply. Get the shapes right and the rest follows.
Write the chain rule yourself
For sigmoid with cross-entropy, the output gradient simplifies to (p - y). Hidden gradients multiply by W2 and tanh's slope.
Gradient check
Compare your gradient with a numerical slope from nudging one weight. If they match, your backprop is right.
XOR proves the hidden layer
A single perceptron can't solve XOR. Your two-layer network can, which shows why hidden layers matter.
See it in code
Walk through the code
1import numpy as np2 3X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]]); y = np.array([[0], [1], [1], [0]]) # XOR4rng = np.random.default_rng(0)5W1, b1 = rng.normal(size=(2, 8)), np.zeros(8)6W2, b2 = rng.normal(size=(8, 1)), np.zeros(1)7sigmoid = lambda z: 1 / (1 + np.exp(-z))8 9for step in range(5000):10 h = np.tanh(X @ W1 + b1) # forward: hidden layer11 p = sigmoid(h @ W2 + b2) # forward: output probability12 dz2 = (p - y) / len(X) # gradient of cross-entropy wrt logits13 dz1 = (dz2 @ W2.T) * (1 - h ** 2) # backward through tanh14 W2 -= 1.0 * h.T @ dz2; b2 -= 1.0 * dz2.sum(0)15 W1 -= 1.0 * X.T @ dz1; b1 -= 1.0 * dz1.sum(0)16print(p.round(2).ravel()) # close to [0, 1, 1, 0]1/4XOR data, random weights for 2 inputs → 8 hidden units → 1 output, and a sigmoid helper.
A complete two-layer network that learns XOR, with no deep learning library. The learning rate here is 1.0.
Save yourself the pain
Common mistakes
Oops
Updating W2 before computing the gradient for the hidden layer.
Do this instead
Compute all gradients first using the current weights, then update. Otherwise layer 1 gets blame from the wrong W2.
Oops
Starting all weights at zero.
Do this instead
Then every hidden unit learns the same thing forever. Use small random values to break the symmetry.
Your mission
Milestone 1: XOR, then MNIST
Quick check
Why must weights start with small random values instead of all zeros?
Go deeper
- 1Implement a 2-layer network with manual forward and backward passes.
- 2Verify your manual gradients against PyTorch's autograd on the same inputs.
- 3Train it on a toy dataset and confirm the loss decreases correctly.
More projects at this stop
Digit Classifier (MNIST)
Stuck on this topic? Ask an Insider
Get 1:1 guidance from people who've walked this exact path — on the InsideEdge app.
