Stop 09 · Deep Learning FundamentalsTopics · 2/10

Activation Functions

In short

Without activation functions, a 100-layer network is no smarter than one straight line. These small functions add the bends that let networks learn real-world patterns.

Non-linear functions (ReLU, sigmoid, tanh) applied after each layer's weighted sum — without them, a neural network would collapse into a single linear function no matter how many layers it has.

The mascot covers its ears happily as a pressure cooker lets out a puff of steam.

Think of it like this

The pressure cooker whistle

A pressure cooker stays quiet while pressure is low. Once it crosses a limit, the whistle blows, and harder pressure means a stronger whistle. ReLU behaves like that: zero below the limit, then it grows with the input.

Watch it happen

Five inputs through different activations

-2

-1

1/5These are raw weighted sums, z, coming into a neuron.

The big ideas

What you need to know

Non-linearity is the point

Stacking linear layers gives another linear layer. Activations let the network bend its decision boundary.

ReLU for hidden layers

Fast and simple, and the usual default. Variants like Leaky ReLU and GELU fix some of its quirks.

Match the output layer

Sigmoid for one yes/no output, softmax for picking one of many classes, nothing for plain regression.

Vanishing gradients

Sigmoid and tanh flatten at the ends, so gradients shrink in deep networks. That's one reason ReLU took over.

See it in code

Walk through the code

python

1import numpy as np2 3z = np.array([-2.0, -1.0, 0.0, 1.0, 2.0])4 5relu    = np.maximum(0, z)6sigmoid = 1 / (1 + np.exp(-z))7tanh    = np.tanh(z)8softmax = np.exp(z) / np.exp(z).sum()9 10print(relu)11print(sigmoid.round(2), tanh.round(2))12print(softmax.round(2))

1/3ReLU: keep positives, zero the rest.

The four activations from the scene, written in one line each with NumPy.

Save yourself the pain

Common mistakes

Oops

Putting softmax on the output of a regression model that predicts prices.

Do this instead

Use no activation (linear) for regression outputs. Softmax is only for picking among classes.

Oops

Using sigmoid in every hidden layer of a deep network.

Do this instead

Use ReLU or GELU in hidden layers. Keep sigmoid for a yes/no output.

Your mission

Plot the activations

~15 min

Quick check

Your network must pick one of 5 flower types. Which activation fits the output layer?

Go deeper

1Plot ReLU, sigmoid, and tanh and compare their shapes.
2Learn why ReLU is the default choice for hidden layers in most modern networks.
3Understand the 'vanishing gradient' problem and why it motivated ReLU's popularity over sigmoid.