Stop 06 · Machine Learning FundamentalsTopics · 2/13

Train / Test / Validation Split

In short

A model that's only tested on data it has already seen will always look brilliant. Splitting your data properly is how you find out if it actually works on new data.

Splitting data so a model is trained on one portion, tuned on a second (validation), and finally evaluated on a completely unseen third (test) — the basic discipline that prevents you from fooling yourself about model performance.

The mascot sits at a study desk with a textbook, a mock test paper and a sealed exam envelope.

Think of it like this

Board exam preparation

You practise textbook exercises (training data). You take mock tests to choose your study strategy (validation data). Then you sit the real board paper you've never seen (test data). If you'd seen the board paper in advance, your marks would prove nothing.

Watch it happen

Ten rows, split three ways

1/4Shuffle the rows first so each split gets a fair mix.

The big ideas

What you need to know

Train to learn

The model fits its parameters only on training rows.

Validate to choose

Use validation data to pick between models and settings. Cross-validation can replace a fixed validation set.

Test once, at the end

If you keep tweaking based on the test score, it stops being an honest estimate.

Time data splits by time

For forecasts, train on the past and test on later dates. Random shuffling would leak the future.

See it in code

Walk through the code

python

1from sklearn.datasets import load_breast_cancer2from sklearn.model_selection import train_test_split3 4X, y = load_breast_cancer(return_X_y=True)5 6X_temp, X_test, y_temp, y_test = train_test_split(7    X, y, test_size=0.2, stratify=y, random_state=42)8X_train, X_val, y_train, y_val = train_test_split(9    X_temp, y_temp, test_size=0.25, stratify=y_temp, random_state=42)10 11print(len(X_train), len(X_val), len(X_test))   # about 60% / 20% / 20%

1/4569 tumour records with a benign or malignant label.

Two calls to train_test_split give three sets. stratify keeps the class mix the same in each.

Save yourself the pain

Common mistakes

Oops

Cleaning, scaling or selecting features on the full dataset before splitting.

Do this instead

Split first. Fit every preprocessing step on training data only, ideally inside a Pipeline.

Oops

Shuffling time series data randomly before splitting.

Do this instead

Split by date so the model never trains on rows from after the test period.

Your mission

Catch yourself cheating

~20 min

Quick check

You tried 30 model settings and picked the one with the best test score. What's the problem?

Go deeper

1Practice a train/test split and explain why test data must never touch the training process.
2Learn why validation data exists separately from test data (for tuning, not final evaluation).
3Understand data leakage — when test information accidentally influences training.