Stop 07 · Deep Learning FoundationsTopics · 5/11

Loss Functions

In short

A network can only improve if it knows how wrong it is. The loss function turns 'wrong' into a single number, and which formula you pick changes what the model cares about.

Functions (MSE, cross-entropy) that quantify how wrong a model's predictions are — the single number gradient descent is trying to minimize.

The blob on a quiz show stage nervously presses a big answer button while a quizmaster watches.

Think of it like this

Negative marking in a quiz

In a quiz with negative marking, saying 'I'm 100% sure it's B' and being wrong hurts far more than a cautious guess. Cross-entropy loss works the same way: being confidently wrong costs a lot, being unsure costs a little.

Watch it happen

Cross-entropy vs confidence

0.9

0.6

0.3

0.05

1/4These are four probabilities the model gave to the correct class, from very sure to almost dismissing it.

The big ideas

What you need to know

MSE for numbers

Mean squared error averages (prediction - truth)². Squaring makes big misses cost much more than small ones.

MAE shrugs off outliers

Mean absolute error averages the plain gaps. One wild row won't dominate it the way it dominates MSE.

Cross-entropy for classes

It punishes low probability on the correct class. It's the standard loss for classification, from spam filters to LLMs.

Logits go in, not probabilities

PyTorch's CrossEntropyLoss applies softmax for you. Pass raw scores (logits), not outputs you've already softmaxed.

See it in code

Walk through the code

python

1import torch2import torch.nn as nn3 4# Regression: predicted vs actual house prices (in lakhs)5pred = torch.tensor([52.0, 75.0, 40.0])6true = torch.tensor([50.0, 80.0, 41.0])7print(nn.MSELoss()(pred, true))   # (4 + 25 + 1) / 3 = 10.08 9# Classification: raw scores (logits) for 3 classes, true class = 010logits = torch.tensor([[2.0, 0.5, -1.0]])11target = torch.tensor([0])12print(nn.CrossEntropyLoss()(logits, target))  # about 0.24, class 0 scored highest

1/3Squared gaps are 4, 25 and 1. Their mean is 10. The 5-lakh miss dominates.

Use MSE when predicting a number and cross-entropy when predicting a class. PyTorch has both ready to call.

Save yourself the pain

Common mistakes

Oops

Applying softmax yourself and then passing the result to CrossEntropyLoss.

Do this instead

CrossEntropyLoss already includes log-softmax. Pass raw logits, or you squash twice and training slows down.

Oops

Using MSE for a classification problem.

Do this instead

MSE gives weak gradients when a classifier is confidently wrong. Cross-entropy trains classifiers faster and more reliably.

Your mission

Feel the loss

~15 min

Quick check

A classifier gives the correct class a probability of 0.01. What does cross-entropy loss do?

Go deeper

1Learn when to use MSE (regression) vs. cross-entropy (classification).
2Implement both loss functions from scratch in NumPy.
3Understand how loss shape affects how easily gradient descent can optimize it.