Stop 05 · Machine Learning FundamentalsTopics · 12/12

Cross-Validation

In short

One train-test split can be lucky or unlucky. Cross-validation tests a model on several different splits, so you get a score you can actually trust.

Splitting data into multiple folds and rotating which fold is used for validation — a more robust way to estimate how a model will generalize than a single train/test split.

The mascot judge tastes food from five plates lined up on a long table.

Think of it like this

A fair cooking trial

Judging a new cook by one dish is luck: maybe it was their best one. So you ask for five different meals on five days and average the marks. Cross-validation does that for models, testing them on five different slices of data.

Watch it happen

5-fold cross-validation

Fold 1

Fold 2

Fold 3

Fold 4

Fold 5

1/5Shuffle the data and cut it into five equal folds.

The big ideas

What you need to know

Every row tested once

Each fold takes a turn as test data, so the score uses all your data efficiently.

Mean and spread

Report the average and the standard deviation. Two models within the spread may be equally good.

Stratify for classes

StratifiedKFold keeps the class balance the same in every fold, which matters for imbalanced data.

Special cases

Use TimeSeriesSplit for time data and GroupKFold when one customer has many rows.

See it in code

Walk through the code

python

1from sklearn.datasets import load_breast_cancer2from sklearn.model_selection import StratifiedKFold, cross_val_score3from sklearn.pipeline import make_pipeline4from sklearn.preprocessing import StandardScaler5from sklearn.linear_model import LogisticRegression6 7X, y = load_breast_cancer(return_X_y=True)8model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))9 10cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)11scores = cross_val_score(model, X, y, cv=cv, scoring="accuracy")12print(scores.round(3), scores.mean().round(3), scores.std().round(3))

1/3Load data and build a pipeline of scaling plus logistic regression.

Putting the scaler inside a pipeline means it's refitted on each training fold, so no fold leaks into another.

Save yourself the pain

Common mistakes

Oops

Scaling or selecting features on all data, then cross-validating.

Do this instead

Wrap preprocessing in a Pipeline so it's learned inside each training fold only.

Oops

Using plain KFold when one patient or customer has many rows.

Do this instead

Use GroupKFold so the same person never appears in both training and test folds.

Your mission

Lucky split vs fair score

~20 min

Quick check

Model A scores 0.91 ± 0.03 in cross-validation, model B scores 0.92 ± 0.04. What's a fair conclusion?

Go deeper

1Implement K-fold cross-validation with scikit-learn's `cross_val_score`.
2Learn stratified K-fold for classification and time-series-aware splitting for temporal data.
3Use cross-validation results to compare two candidate models fairly.