Stop 05 · Machine Learning FundamentalsTopics · 4/12

Decision Trees

In short

Decision trees learn a set of yes/no questions from data. They handle messy features, need little preparation and are easy to draw for a manager, which makes them a great model to understand deeply.

A model that splits data on feature thresholds to form a tree of decisions — interpretable, handles non-linear relationships, and the building block of Random Forests and XGBoost.

The mascot stands at a fork in a garden path that splits again into smaller paths.

Think of it like this

Signposts on a trek

At each fork, a signpost asks one question. 'Tired? Take the short trail.' 'Raining? Take the covered path.' After a few forks you reach a final spot. A decision tree is a chain of such questions, ending in a prediction.

Watch it happen

A tiny loan approval tree

Income > 50k?

Credit > 700?

Guarantor?

Approve

Review

Approve small

Decline

1/4The root question is the split that best separates good and bad loans in the training data.

The big ideas

What you need to know

Greedy splits

At each node, the tree picks the feature and threshold that make the groups purest, measured by Gini or entropy.

Easy to explain

You can print or plot the rules. Great for showing why a prediction was made.

Overfits easily

Grown fully, a tree memorises noise. Limit max_depth or min_samples_leaf.

Building block for forests

Random forests and gradient boosting combine many trees to fix a single tree's weaknesses.

See it in code

Walk through the code

python

1from sklearn.datasets import load_iris2from sklearn.tree import DecisionTreeClassifier, export_text3 4iris = load_iris()5tree = DecisionTreeClassifier(max_depth=2, random_state=0)6tree.fit(iris.data, iris.target)7 8print(export_text(tree, feature_names=list(iris.feature_names)))9print(tree.feature_importances_.round(2))

1/3A shallow tree with at most two questions per path.

A depth-2 tree prints as a handful of readable if-else rules. Feature importances show which measurements it relied on.

Save yourself the pain

Common mistakes

Oops

Growing a tree with no depth limit and trusting its training score.

Do this instead

Limit depth or leaf size, and judge it with cross-validation.

Oops

Assuming a feature unused by the tree is useless.

Do this instead

A correlated feature may have been picked instead. Importance depends on what else is in the data.

Your mission

Draw your own tree

~25 min

Quick check

Why does a fully grown decision tree often score 100% on training data but worse on test data?

Go deeper

1Train a decision tree and visualize it to see exactly how it splits data.
2Learn how splits are chosen (Gini impurity, entropy).
3Experiment with max depth to see overfitting happen in real time.