Stop 06 · Machine Learning FundamentalsTopics · 3/13

Overfitting & Underfitting

In short

The most common reason models fail in real life isn't a bad algorithm. It's a model that memorised its training data, or one too simple to learn anything useful.

Overfitting means a model memorized training data noise and fails on new data; underfitting means it's too simple to capture real patterns even in training data — the two failure modes every model sits between.

The mascot memorises from a thick book with its eyes shut, a tall stack of old question papers beside it.

Think of it like this

Two kinds of exam prep

One student mugs up last year's answers word for word, then panics when the questions are reworded. That's overfitting. Another only reads the chapter titles and can't answer anything. That's underfitting. The topper understands the ideas and handles new questions.

Watch it happen

Decision tree depth vs accuracy (train / validation %)

d1 90/88

d3 96/93

d5 98/94

d10 100/91

d∞ 100/90

1/4Each cell is one tree depth, with its training and validation accuracy. These numbers are illustrative.

The big ideas

What you need to know

Underfitting

Poor scores on both training and validation. Try a more flexible model or better features.

Overfitting

Great training score, much worse validation score. The gap between them is the warning sign.

Ways to fight overfitting

More data, simpler models, regularisation, limiting tree depth, early stopping and dropping noisy features.

Plot learning curves

Scores against training size or complexity show which problem you have.

See it in code

Walk through the code

python

1from sklearn.datasets import load_breast_cancer2from sklearn.model_selection import train_test_split3from sklearn.tree import DecisionTreeClassifier4 5X, y = load_breast_cancer(return_X_y=True)6X_tr, X_val, y_tr, y_val = train_test_split(X, y, random_state=0)7 8for depth in [1, 3, 5, 10, None]:9    m = DecisionTreeClassifier(max_depth=depth, random_state=0).fit(X_tr, y_tr)10    print(depth, round(m.score(X_tr, y_tr), 3), round(m.score(X_val, y_val), 3))

1/3Load data and hold out a validation set.

Print training and validation accuracy side by side for each depth, and watch the gap open up.

Save yourself the pain

Common mistakes

Oops

Celebrating a 100% training accuracy.

Do this instead

Training accuracy says little. Judge models on validation or cross-validation scores.

Oops

Adding more and more features to fix a model that already overfits.

Do this instead

More features usually make overfitting worse. Simplify, regularise or get more rows.

Your mission

Draw the sweet spot

~25 min

Quick check

Training accuracy is 99%, validation accuracy is 72%. What's happening?

Go deeper

1Plot training vs validation error as model complexity increases, and identify both zones.
2Learn the common symptoms: overfitting shows a big train/test performance gap.
3Practice deliberately overfitting a tiny model, then simplifying it back down.