Stop 06 · Machine Learning FundamentalsTopics · 9/13

Random Forests

In short

One decision tree is jumpy. Hundreds of slightly different trees voting together are steady and accurate. Random forests are a reliable default for tabular data, often with little tuning.

An ensemble of many decision trees trained on random subsets of data and features, with predictions averaged (or voted) together — usually far more accurate and stable than any single tree.

The mascot sits at a long judges' table facing a fest stage, with a row of blank scorecards.

Think of it like this

A panel of fest judges

At a college fest, each judge watched only some of the performances, from a different seat. Alone, any one judge might be biased. Average all their scores and you get a fair result. A random forest is that panel of judges, made of trees.

Watch it happen

Five trees vote on one loan application

T1: repay

T2: default

T3: repay

T4: repay

T5: default

1/4Each tree was trained on a different random sample of rows, using random subsets of features at each split.

The big ideas

What you need to know

Bagging

Each tree trains on a bootstrap sample: rows drawn at random with replacement.

Random features

Each split considers only a random subset of features, so trees don't all look alike.

Averaging cuts variance

Individual errors cancel out, so the forest is far steadier than any single tree.

Free validation score

Rows a tree didn't see give an out-of-bag score, a built-in estimate of test accuracy.

See it in code

Walk through the code

python

1from sklearn.datasets import load_breast_cancer2from sklearn.model_selection import train_test_split3from sklearn.ensemble import RandomForestClassifier4 5X, y = load_breast_cancer(return_X_y=True)6X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=0)7 8rf = RandomForestClassifier(n_estimators=300, max_features="sqrt",9                            oob_score=True, n_jobs=-1, random_state=0)10rf.fit(X_tr, y_tr)11print("OOB:", round(rf.oob_score_, 3), "Test:", round(rf.score(X_te, y_te), 3))

1/3300 trees, random feature subsets, out-of-bag scoring on, using all CPU cores.

300 trees, each split looking at a random square-root share of features. The out-of-bag score is usually close to the test score.

Save yourself the pain

Common mistakes

Oops

Expecting a random forest to predict beyond the range of training values.

Do this instead

Forests average training targets, so they can't extrapolate. A house bigger than any seen gets a capped price.

Oops

Trusting default feature importances blindly.

Do this instead

They can favour features with many unique values. Check with permutation importance on validation data.

Your mission

One tree vs a forest

~20 min

Quick check

Why do random forests use random feature subsets at each split?

Go deeper

1Fit a random forest and compare its performance to a single decision tree.
2Learn how feature importance scores work in a random forest.
3Understand why averaging many 'weak' trees reduces overfitting (the wisdom of crowds effect).