Stop 06 · Machine Learning FundamentalsTopics · 4/13

Bias-Variance Tradeoff

In short

Every model's error comes partly from being too rigid and partly from being too jumpy. Understanding this tradeoff tells you whether to make a model simpler or more flexible.

Bias is error from overly simplistic assumptions (underfitting), variance is error from being overly sensitive to training data specifics (overfitting) — total error is a tradeoff between the two.

The mascot throws darts at a balloon board at a fair, with the darts bunched off to one side.

Think of it like this

Darts at the mela

One player's darts all land together, but always in the same wrong corner: high bias. Another's darts are centred on average but scattered all over the board: high variance. You want darts that land close together and near the bullseye.

Watch it happen

Two ways a model can miss

High bias

Too simple, misses the real pattern
Similar errors on train and test
Fix: richer model or features

High variance

Too flexible, chases noise
Changes a lot with new data
Fix: more data or regularise

Make the model more flexible until validation error stops improving, then stop. That point balances bias and variance.

The big ideas

What you need to know

Bias: wrong assumptions

A straight line fitted to a curved pattern is always a bit off, however much data you add.

Variance: too sensitive

A very wiggly model would look different on every new sample of data.

The tradeoff

More flexibility lowers bias but raises variance. Total error is lowest somewhere in between.

Tools that shift it

Regularisation and bagging cut variance. Boosting and extra features cut bias.

See it in code

Walk through the code

python

1import numpy as np2from sklearn.pipeline import make_pipeline3from sklearn.preprocessing import PolynomialFeatures4from sklearn.linear_model import LinearRegression5from sklearn.model_selection import cross_val_score6 7rng = np.random.default_rng(0)8x = rng.uniform(0, 6, 40).reshape(-1, 1)9y = np.sin(x).ravel() + rng.normal(0, 0.2, 40)10 11for degree in [1, 4, 15]:12    model = make_pipeline(PolynomialFeatures(degree), LinearRegression())13    mse = -cross_val_score(model, x, y, cv=5, scoring="neg_mean_squared_error").mean()14    print(degree, round(mse, 3))

1/3Make 40 noisy points that follow a sine curve.

Degree 1 is too rigid (high bias), degree 15 is too wiggly (high variance). The middle degree usually wins on cross-validated error.

Save yourself the pain

Common mistakes

Oops

Thinking more complex models are always better.

Do this instead

Past a point, extra flexibility adds variance and validation error goes up.

Oops

Trying to fix high bias by collecting more data.

Do this instead

More data mainly helps variance. For high bias, give the model more flexibility or better features.

Your mission

See the U-shape

~25 min

Quick check

Your model's training and validation errors are both high and close together. What's the likely issue?

Go deeper

1Sketch the classic bias-variance tradeoff curve and label both zones.
2Connect a high-bias and a high-variance model type to real algorithms you'll learn.
3Practice explaining this tradeoff to someone with no ML background.