Stop 08 · Data Cleaning & WranglingTopics · 5/9

Outlier Handling

In short

One ₹50 lakh order in a sea of ₹500 orders can wreck your average. But it might be a typo, a bulk buyer or fraud. Outliers need investigating before anyone deletes them.

Applying Phase 4's outlier detection with a clear action plan — investigate, cap, remove, or keep and flag — and always documenting the decision, since outlier handling silently changes your conclusions.

Think of it like this

One giant sunflower

A row of small marigolds and one sunflower towering over them. Is it a different plant someone planted on purpose, a fluke, or did a kid stick a fake one there? You find out before you pull it out. Outliers deserve the same curiosity.

Watch it happen

Finding outliers with the IQR rule

180

1/4Delivery times in minutes, sorted. One value looks very different.

The big ideas

What you need to know

Investigate first

Look at the full row. Typos, test orders and unit mix-ups have tell-tale signs.

IQR and z-score

IQR fences work for skewed data. Z-scores beyond about 3 suit roughly normal data.

Options, not one rule

Correct it, remove it, cap it at a percentile, or keep it and use the median instead of the mean.

Sometimes the outlier is the story

A fraud spike or a viral day isn't noise. It might be the most important row in the data.

See it in code

Walk through the code

python

1q1, q3 = df["minutes"].quantile([0.25, 0.75])2iqr = q3 - q13low, high = q1 - 1.5 * iqr, q3 + 1.5 * iqr4 5df["is_outlier"] = ~df["minutes"].between(low, high)6print(df.loc[df["is_outlier"], ["order_id", "minutes", "city"]])7 8cap = df["minutes"].quantile(0.99)9df["minutes_capped"] = df["minutes"].clip(upper=cap)10 11print(df["minutes"].mean(), df["minutes_capped"].mean(), df["minutes"].median())

1/4Compute the quartiles and the two fences.

Flag with IQR fences, inspect the flagged rows, then compare mean, capped mean and median before choosing.

Save yourself the pain

Common mistakes

Oops

Deleting everything outside the IQR fences automatically.

Do this instead

Flag, investigate, then decide per case. Write down what you did and why.

Oops

Using the mean on data with big outliers and reporting it as 'typical'.

Do this instead

Report the median for typical values, or show both with a note on the extremes.

Your mission

Hunt one outlier

~25 min

Quick check

Q1 = 40 and Q3 = 60. Using the 1.5 × IQR rule, what's the upper fence?

Go deeper

1Take a dataset with known outliers and decide + justify a handling strategy for each.
2Practice capping (winsorizing) values instead of just deleting them, as an alternative.
3Re-run a summary statistic before and after outlier handling to see the impact.