Stop 07 · Model Evaluation & TuningTopics · 7/8

Ensemble Methods

In short

One decision tree is a bit moody. Hundreds of them voting together are steady and strong. Ensembles power many winning models on tabular data.

Combining multiple models' predictions to improve accuracy and robustness — bagging (random forests), boosting (XGBoost), and stacking (combining different model types) are the three main families.

The mascot holds a microphone on a quiz show stage while the audience raises hands to vote.

Think of it like this

The quiz show audience poll

On a TV quiz show, one audience member might be wrong, but the majority of a hall full of people is usually right. Their mistakes are random and cancel out. An ensemble asks many models and combines their answers the same way.

Watch it happen

Three families of ensembles

Ensembles

Bagging

Boosting

Stacking

Random Forest

Grad. boosting

XGBoost etc.

Meta-model

1/4All ensembles combine several models. They differ in how the models are trained and joined.

The big ideas

What you need to know

Bagging cuts variance

Random samples make each tree a bit different. Averaging them smooths out the noise one tree picks up.

Boosting cuts bias

Each new tree focuses on what's still wrong. Powerful, but it can overfit if you add too many rounds.

Diversity is the point

Combining models that make the same mistakes gains nothing. Different errors are what cancel out.

Harder to explain

Three hundred trees are not easy to read. Use feature importance or SHAP to explain predictions.

See it in code

Walk through the code

python

1from sklearn.datasets import load_breast_cancer2from sklearn.ensemble import GradientBoostingClassifier, RandomForestClassifier, VotingClassifier3from sklearn.linear_model import LogisticRegression4from sklearn.model_selection import cross_val_score5from sklearn.pipeline import make_pipeline6from sklearn.preprocessing import StandardScaler7 8X, y = load_breast_cancer(return_X_y=True)9lr = make_pipeline(StandardScaler(), LogisticRegression())10rf = RandomForestClassifier(n_estimators=300, random_state=0)11gb = GradientBoostingClassifier(random_state=0)12vote = VotingClassifier([("lr", lr), ("rf", rf), ("gb", gb)], voting="soft")13 14for name, m in [("forest", rf), ("boosting", gb), ("vote", vote)]:15    print(name, cross_val_score(m, X, y, cv=5).mean().round(3))

1/4Random forest: 300 trees on bootstrapped samples. That's bagging.

A bagging model, a boosting model and a soft vote across three different model types, all scored with 5-fold cross-validation.

Save yourself the pain

Common mistakes

Oops

Adding boosting rounds forever because training accuracy keeps rising.

Do this instead

Watch validation scores, use early stopping, and tune learning rate together with the number of trees.

Oops

Voting across three nearly identical models and expecting a big jump.

Do this instead

Mix model types or feature views so their errors differ.

Your mission

One tree vs a forest

~20 min

Quick check

What is the main difference between bagging and boosting?

Go deeper

1Compare bagging and boosting conceptually — what error each is designed to reduce.
2Try a simple stacking approach: combine predictions from 2-3 different model types.
3Learn why ensembles tend to generalize better than any single strong model.