Stop 06 · Machine Learning FundamentalsTopics · 10/13

Gradient Boosting (XGBoost, LightGBM)

In short

On tabular data like transactions or customer records, gradient boosting is very often the most accurate model. XGBoost and LightGBM are the libraries most teams reach for.

An ensemble method that builds trees sequentially, each one correcting the errors of the previous — XGBoost and LightGBM are the industry-standard implementations, often winning tabular-data competitions.

The mascot pins corrected test papers on a wall, each newer one with fewer red marks.

Think of it like this

Fixing last test's mistakes

After each practice test, a smart student studies only the questions they got wrong. The next test, fewer mistakes. Then they fix those. Gradient boosting builds trees the same way: each new small tree focuses on what the previous ones got wrong.

Watch it happen

Predicting a ₹100 price, round by round

1/5Round 0: start with a simple guess, the average price, ₹70. The error left over is ₹30.

The big ideas

What you need to know

Trees in sequence

Unlike a forest's independent trees, each boosting tree corrects the errors of all the trees before it.

Learning rate and tree count

A smaller learning rate needs more trees but often generalises better.

Early stopping

Watch a validation set and stop adding trees once it stops improving. It prevents overfitting.

XGBoost, LightGBM, CatBoost

Popular, fast implementations. LightGBM is known for speed on big data, CatBoost for categorical features.

See it in code

Walk through the code

python

1from sklearn.datasets import load_breast_cancer2from sklearn.model_selection import train_test_split3from xgboost import XGBClassifier4 5X, y = load_breast_cancer(return_X_y=True)6X_tr, X_val, y_tr, y_val = train_test_split(X, y, random_state=0)7 8model = XGBClassifier(n_estimators=1000, learning_rate=0.05, max_depth=4,9                      early_stopping_rounds=50, eval_metric="logloss")10model.fit(X_tr, y_tr, eval_set=[(X_val, y_val)], verbose=False)11print(model.best_iteration, round(model.score(X_val, y_val), 3))

1/3Small trees, a slow learning rate and early stopping after 50 rounds without improvement.

Up to 1,000 small trees, stopping once validation loss hasn't improved for 50 rounds. Keep a separate test set for the final score.

Save yourself the pain

Common mistakes

Oops

Using a high learning rate with thousands of trees and no early stopping.

Do this instead

Use a small learning rate with early stopping on a validation set.

Oops

Reporting the early-stopping validation score as your final result.

Do this instead

That set helped choose the tree count. Report the score on a separate, untouched test set.

Your mission

Boost vs forest

~30 min

Quick check

What's the main difference between a random forest and gradient boosting?

Go deeper

1Fit an XGBoost or LightGBM model and compare it to your random forest baseline.
2Learn the key hyperparameters: learning rate, number of estimators, max depth.
3Understand at a high level why boosting often outperforms bagging (random forests) on tabular data.