Stop 06 · Feature Engineering & Model EvaluationTopics · 12/12

Overfitting & Regularization (L1/L2)

In short

A model that fits every wiggle of the training data will trip over new data. Regularisation is a gentle tax on complexity that keeps it from showing off.

Overfitting is memorizing training data instead of learning general patterns; L1 (Lasso) and L2 (Ridge) regularization penalize large weights to keep models simpler and more general.

The blob as a tailor holds a measuring tape around a mannequin wearing a very tight kurta.

Think of it like this

The over-eager tailor

A tailor measures you right after lunch and stitches a kurta that hugs every fold. Next morning, it doesn't fit. A wise tailor leaves a little ease. Regularisation tells the model the same thing: fit the shape, not every wrinkle.

Watch it happen

What happens to the weights

4.2

-3.1

2.5

0.8

0.3

-0.2

1/4Six weights from a plain linear model. Some are big, some tiny. The tiny ones may just be fitting noise.

The big ideas

What you need to know

Spot overfitting by the gap

High training score, much lower validation score. The model learnt the training rows, not the pattern.

L2 adds squared weights

Ridge adds alpha times the sum of squared weights to the loss. Big weights get expensive, so they shrink.

L1 adds absolute weights

Lasso adds alpha times the sum of absolute weights. It tends to set unhelpful weights exactly to zero.

alpha sets the strength

Bigger alpha means a stronger penalty and a simpler model. Too big, and it underfits. Tune it with cross-validation.

See it in code

Walk through the code

python

1import numpy as np2from sklearn.datasets import make_regression3from sklearn.linear_model import LinearRegression, Ridge, Lasso4 5# 20 features, but only 5 actually matter6X, y = make_regression(n_samples=60, n_features=20, n_informative=5, noise=10, random_state=0)7 8for model in [LinearRegression(), Ridge(alpha=10), Lasso(alpha=1)]:9    model.fit(X, y)10    zeros = np.sum(np.abs(model.coef_) < 1e-6)11    print(type(model).__name__, "zero weights:", zeros)

1/3Fake data where only 5 of 20 features drive y. The rest are noise the model might overfit.

Plain regression uses all 20 features. Ridge shrinks them. Lasso sets many of the useless ones to exactly zero.

Save yourself the pain

Common mistakes

Oops

Applying Ridge or Lasso without scaling features first.

Do this instead

The penalty treats all weights equally, so features in big units get unfairly shrunk. Scale first.

Oops

Picking alpha by checking which gives the best test score.

Do this instead

Use RidgeCV, LassoCV or GridSearchCV on training data. Keep the test set for the final check.

Your mission

Tune the penalty

~30 min

Quick check

You want a model that automatically ignores useless features. Which regularisation helps most?

Go deeper

1Deliberately overfit a model on a small dataset to see the symptom clearly.
2Apply L1 and L2 regularization and compare their effect on model coefficients.
3Learn why L1 tends to zero out weights (feature selection) while L2 shrinks them smoothly.