Stop 06 · Feature Engineering & Model EvaluationTopics · 6/12

Ensemble Methods (Bagging, Boosting)

In short

One decision tree is a bit moody. Hundreds of trees working together are steady and sharp. Ensembles are why tree models still win so many problems on tabular data.

Combining multiple weaker models into a stronger one — bagging trains models in parallel on random subsets (Random Forest), boosting trains sequentially, each correcting the last (XGBoost).

The blob sits on a panel of judges holding up score cards, next to a teacher marking a test.

Think of it like this

Judges' panel vs a tuition teacher

Bagging is a dance competition panel: many judges score on their own, and the average is fairer than any one judge. Boosting is a tuition teacher who checks your last test and sets the next one only on topics you got wrong.

Watch it happen

Two ways to combine trees

Bagging

Trees train in parallel
Each sees a random sample
Averaging cuts variance

Boosting

Trees train one after another
Each fixes the last one's errors
Lowers bias, can overfit

Random forest (bagging) is a safe, low-effort default. Gradient boosting usually scores higher on tabular data but needs more tuning.

The big ideas

What you need to know

Bagging = bootstrap + average

Each tree trains on a random sample of rows drawn with replacement. Predictions are averaged or voted on.

Random forest adds a twist

Each split also looks at only a random subset of features, so the trees disagree more and the average gets better.

Boosting learns from mistakes

Gradient boosting fits each new small tree to the errors left by all the previous trees, then adds it with a small weight.

Learning rate vs number of trees

A smaller learning rate needs more trees but usually generalises better. Tune these two together.

How it works

One round of gradient boosting

1
Start with a simple guess
For example, predict the average value for everyone.
2
Measure what's left
Compute the errors (residuals) between the guess and the true answers.
3
Fit a small tree to the errors
This tree learns where the current model goes wrong.
4
Add a fraction of it
Scale the tree by the learning rate and add it to the model. Repeat.

See it in code

Walk through the code

python

1from sklearn.datasets import load_breast_cancer2from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier3from sklearn.model_selection import cross_val_score4 5X, y = load_breast_cancer(return_X_y=True)6 7models = {8    "bagging (random forest)": RandomForestClassifier(n_estimators=300, random_state=0),9    "boosting (gradient boosting)": GradientBoostingClassifier(random_state=0),10}11for name, model in models.items():12    scores = cross_val_score(model, X, y, cv=5)13    print(f"{name}: {scores.mean():.3f}")

1/2A random forest of 300 trees, and a gradient boosting model with its default 100 small trees.

Both ensembles plug into the same scikit-learn API. Cross-validation gives each a fair score on the same folds.

Save yourself the pain

Common mistakes

Oops

Adding more and more boosting rounds because training accuracy keeps rising.

Do this instead

Boosting can overfit. Watch a validation score and stop when it stops improving.

Oops

Expecting bagging to fix a model that's too simple.

Do this instead

Averaging reduces variance, not bias. If each tree underfits, the average will too. That's where boosting helps.

Your mission

Race the ensembles

~25 min

Quick check

What is the main difference between how bagging and boosting build their trees?

Go deeper

1Compare a bagging model (Random Forest) and a boosting model (XGBoost) on the same dataset.
2Learn conceptually why boosting often outperforms bagging on tabular data.
3Understand the added risk of overfitting with boosting and how early stopping mitigates it.