Activation Functions

Summary

Non-linear functions (ReLU, sigmoid, tanh) applied after each layer's weighted sum — without them, a neural network would collapse into a single linear function no matter how many layers it has.

How to Learn This

1Plot ReLU, sigmoid, and tanh and compare their shapes.
2Learn why ReLU is the default choice for hidden layers in most modern networks.
3Understand the 'vanishing gradient' problem and why it motivated ReLU's popularity over sigmoid.

Neural Network Basics

Backpropagation & Gradient Descent

More topics in Deep Learning Fundamentals

Neural Network Basics Backpropagation & Gradient Descent Building Networks With TensorFlow / Keras Building Networks With PyTorch Convolutional Neural Networks (CNNs)Recurrent Neural Networks (RNNs) & LSTMs Transfer Learning Regularization in Deep Learning (Dropout, Batch Norm)GPUs & Training Basics

Stuck on this topic? Ask an Insider

Get 1:1 guidance from people who've walked this exact path — free on the InsideEdge app.

Download