Stop 07 · Model Evaluation & TuningTopics · 6/8

Regularization (L1 / L2)

In short

A model that memorises the training data scores brilliantly there and flops on new data. Regularization adds a penalty for complexity, so the model keeps only what really helps.

Adding a penalty for large model weights to reduce overfitting — L1 (Lasso) can shrink some weights to exactly zero (feature selection), L2 (Ridge) shrinks all weights smoothly toward zero.

The mascot sits on an overstuffed suitcase next to a hanging luggage scale, with extra clothes on the floor.

Think of it like this

The airline weight limit

Packing for a flight with a 15 kg limit forces choices. L1 is like leaving some items at home completely: the extra shoes, the third jacket. L2 is like packing lighter versions of everything: smaller bottles, thinner clothes. Either way, the bag carries only what matters.

Watch it happen

Five coefficients under two penalties (example values)

4.2

-3.1

0.8

0.3

-0.2

1/4A plain linear regression gives these weights. The last three features add little but still get a say.

The big ideas

What you need to know

L1 can drop features

Lasso adds the sum of absolute weights to the loss. Many weights end up exactly zero, a built-in feature selection.

L2 shrinks smoothly

Ridge adds the sum of squared weights. It handles correlated features well and keeps them all small.

Alpha is the knob

In scikit-learn, alpha sets penalty strength for Ridge and Lasso. For LogisticRegression, C is the inverse: smaller C, stronger penalty.

Scale features first

The penalty treats all weights alike, so features measured in big units get unfairly punished unless you standardise.

See it in code

Walk through the code

python

1from sklearn.datasets import load_diabetes2from sklearn.linear_model import Lasso, LinearRegression, Ridge3 4X, y = load_diabetes(return_X_y=True)5 6for model in [LinearRegression(), Ridge(alpha=1.0), Lasso(alpha=1.0)]:7    model.fit(X, y)8    zeros = (model.coef_ == 0).sum()9    print(type(model).__name__, model.coef_.round(1), "zero coefs:", zeros)

1/3Ten medical features, already standardised, predicting disease progression.

The diabetes features come pre-scaled. Ridge shrinks the weights, and Lasso sets several of them to exactly zero.

Save yourself the pain

Common mistakes

Oops

Applying Ridge or Lasso to unscaled features like salary in rupees next to age in years.

Do this instead

Put StandardScaler and the model in one Pipeline so every feature is penalised fairly.

Oops

Picking alpha by eye and reusing it everywhere.

Do this instead

Use RidgeCV, LassoCV or a grid search to choose alpha for each dataset.

Your mission

Trace the Lasso path

~25 min

Quick check

You have 200 features and suspect only a handful matter. Which penalty helps you find them?

Go deeper

1Fit a regularized regression model and compare it to an unregularized version.
2Learn why L1 regularization can effectively remove unimportant features.
3Experiment with different regularization strengths and observe the effect on overfitting.