Stop 07 · Deep Learning FoundationsTopics · 2/11

Perceptrons

In short

Every giant neural network is built from one tiny decision maker repeated millions of times. Meet the perceptron: multiply, add, decide. That's the whole trick.

The simplest neural unit — a weighted sum passed through a step function — historically the first neural network model and still the conceptual starting point for understanding layers.

The blob as a hostel warden weighs three small objects on a scale before handing over a pass.

Think of it like this

The hostel warden's late pass

The warden weighs a few things: did your parents call (counts a lot), is it exam week (counts against you), your past record (counts a little). If the total crosses her mental line, you get the pass. A perceptron does exactly this with numbers.

Watch it happen

Inside one perceptron

Inputs

Weights

Sum + bias

Step

Output

1/5Inputs arrive as numbers, for example x1 = 1 (parents called), x2 = 0 (not exam week).

The big ideas

What you need to know

Weighted sum, then decide

Output = step(w1·x1 + w2·x2 + b). The weights and bias are the only things that get learnt.

Learns from mistakes

The perceptron rule changes weights only when a prediction is wrong, by error times input times learning rate.

Draws one straight line

A single perceptron can only separate classes with a straight line, so it handles AND and OR but not XOR.

Stack them to go further

Put perceptron-like units in layers with smooth activations and you get a neural network that can bend that line.

See it in code

Walk through the code

python

1import numpy as np2 3X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])4y = np.array([0, 0, 0, 1])            # logical AND5w, b, lr = np.zeros(2), 0.0, 1.06 7for epoch in range(10):8    for xi, target in zip(X, y):9        pred = int(xi @ w + b > 0)    # weighted sum, then step10        error = target - pred11        w += lr * error * xi          # nudge weights only when wrong12        b += lr * error13 14print(w, b, [int(xi @ w + b > 0) for xi in X])  # [0, 0, 0, 1]

1/4Four input pairs and the AND answers. Weights and bias start at zero.

Ten passes over four rows are enough for a perceptron to learn AND. Try y = [0, 1, 1, 0] (XOR) and watch it never settle.

Save yourself the pain

Common mistakes

Oops

Thinking a perceptron is the same as a modern neuron.

Do this instead

Modern units use smooth activations like ReLU or sigmoid so gradients can flow. The perceptron's hard step can't be trained with backprop.

Oops

Expecting one perceptron to learn any pattern.

Do this instead

It only finds straight-line boundaries. XOR needs at least one hidden layer.

Your mission

Teach it AND, OR, then XOR

~20 min

Quick check

Why can't a single perceptron learn XOR?

Go deeper

1Implement a single perceptron from scratch and train it on a linearly separable dataset.
2Learn why a single perceptron can't solve XOR (the motivation for multi-layer networks).
3Connect the perceptron to logistic regression — they're nearly identical.