Stop 05 · Exploratory Data Analysis & Feature EngineeringTopics · 3/9

Outlier Detection & Treatment

In short

One wrong entry, like a 900-minute delivery, can drag your averages and confuse your model. The skill is spotting outliers, then deciding whether each is an error or a real rare event.

Identifying data points far from the rest — sometimes genuine signal, sometimes an error — and deciding whether to investigate, cap, remove, or leave them, since some models are far more outlier-sensitive than others.

The mascot stares at one giant pumpkin sitting at the end of a row of normal ones.

Think of it like this

The giant pumpkin at the mandi

A row of normal pumpkins, and then one the size of a scooter. Is it a prize-winning pumpkin, or did someone stack two together? You don't throw it out blindly. You check what it is, then decide.

Watch it happen

Delivery times in minutes, IQR rule

1/5Seven delivery times. The mean is about 26, but most deliveries took 12 to 16 minutes.

The big ideas

What you need to know

IQR fences

Values below Q1 − 1.5×IQR or above Q3 + 1.5×IQR are flagged. Works well on skewed data.

Z-scores

How many standard deviations from the mean. Above about 3 is often flagged, but only for roughly normal data.

Investigate before acting

Data entry error? Fix or remove it. Real rare event? Keep it, and maybe model it separately.

Choose a treatment

Remove, cap (winsorise), log-transform, or use models less sensitive to outliers, like trees.

See it in code

Walk through the code

python

1import pandas as pd2 3s = pd.Series([12, 15, 14, 13, 16, 15, 98], name="delivery_min")4q1, q3 = s.quantile([0.25, 0.75])5iqr = q3 - q16low, high = q1 - 1.5 * iqr, q3 + 1.5 * iqr7print(s[(s < low) | (s > high)])         # 988 9capped = s.clip(lower=low, upper=high)   # cap instead of delete10print(capped.max())                      # 18.5

1/4Seven delivery times, one suspicious.

The IQR rule in a few lines, and capping with clip so you keep the row but limit its pull.

Save yourself the pain

Common mistakes

Oops

Deleting every flagged outlier automatically.

Do this instead

Fraud, VIP customers and system failures are often outliers, and often exactly what you need to study.

Oops

Using z-scores on very skewed data like income.

Do this instead

Use the IQR rule, or log-transform first. Skew inflates the standard deviation and hides real outliers.

Your mission

Hunt outliers

~25 min

Quick check

A salary column has a few CEO salaries far above the rest, and they're correct. What's a sensible approach?

Go deeper

1Use a boxplot and the IQR method to flag outliers in a sample dataset.
2Learn which model types (linear regression) are outlier-sensitive versus robust (tree-based models).
3Practice capping (winsorizing) values as an alternative to deleting them.