Stop 06 · Feature Engineering & Model EvaluationTopics · 7/12

XGBoost & LightGBM

In short

When the data is a spreadsheet of rows and columns, these two libraries are often the first thing experienced engineers try. They are gradient boosting, tuned to be fast and accurate.

Industry-standard gradient boosting libraries — usually the strongest models on structured/tabular data and a staple of ML competitions and production systems.

The blob in a chef hat stirs several giant cooking pots at once in a busy catering kitchen.

Think of it like this

Same biryani, wedding-caterer speed

Your mom's biryani recipe and a wedding caterer's are basically the same. The caterer just has huge vessels, prep in batches and a team that knows shortcuts. XGBoost and LightGBM cook the same gradient boosting recipe, at caterer scale.

Watch it happen

Training with early stopping

Bin data

Add tree

Check val

Repeat

Stop early

1/5Feature values are grouped into bins (histograms), so finding the best split is much faster.

The big ideas

What you need to know

Built for speed

Histogram binning, multithreading and optional GPU training let them handle millions of rows quickly.

Missing values handled

Both learn which way to send missing values at each split, so you often don't need to fill blanks first.

LightGBM grows leaf-wise

LightGBM expands the leaf that reduces error most, which is fast but can overfit small data. Limit num_leaves.

A few knobs matter most

learning_rate, number of trees, tree depth or leaves, and subsampling. Early stopping picks the number of trees for you.

See it in code

Walk through the code

python

1from sklearn.datasets import load_breast_cancer2from sklearn.model_selection import train_test_split3from xgboost import XGBClassifier4 5X, y = load_breast_cancer(return_X_y=True)6X_tr, X_val, y_tr, y_val = train_test_split(X, y, stratify=y, random_state=0)7 8model = XGBClassifier(n_estimators=1000, learning_rate=0.05, max_depth=4,9                      early_stopping_rounds=50, eval_metric="logloss")10model.fit(X_tr, y_tr, eval_set=[(X_val, y_val)], verbose=False)11 12print("trees actually used:", model.best_iteration + 1)13print("val accuracy:", model.score(X_val, y_val))14# LightGBM: from lightgbm import LGBMClassifier, same fit/predict idea

1/3A low learning rate, shallow trees, and a rule to stop after 50 rounds without improvement.

Ask for up to 1000 trees and let early stopping decide how many are actually useful. It follows the scikit-learn fit/predict style.

Save yourself the pain

Common mistakes

Oops

Using the test set as the early stopping eval_set.

Do this instead

Early stopping is a tuning decision. Use a separate validation split, and keep the test set for the final score.

Oops

Reaching for XGBoost on images or raw text.

Do this instead

Boosted trees shine on tabular data. Images, audio and text usually need neural networks.

Your mission

Boosting face-off

~30 min

Quick check

You set n_estimators=1000 with early_stopping_rounds=50, and best_iteration comes out as 180. What happened?

Go deeper

1Train an XGBoost or LightGBM model and compare it against scikit-learn's GradientBoosting.
2Learn key hyperparameters: learning rate, max depth, n_estimators, early stopping rounds.
3Use built-in feature importance to interpret a trained boosted model.