Stop 04 · Statistics for AnalystsTopics · 10/11

Outlier Detection

In short

One ₹9,99,999 order in a list of ₹300 orders can wreck your average. Is it fraud, a typo, or your best customer? Finding outliers is easy. Deciding what they mean is the real skill.

Identifying data points that are unusually far from the rest — sometimes genuine signal (a fraud case), sometimes an error (a data entry typo) — and knowing which is which changes what you should do about them.

The jaadu stares at one huge pumpkin sitting in a cart of small tomatoes.

Think of it like this

The giant pumpkin in the tomato cart

A sabziwala's cart is full of small tomatoes and one enormous pumpkin. You'd notice it instantly. Did it land there by mistake, or is it a prize pumpkin someone ordered? You check before you throw it out. Treat data outliers the same way.

Watch it happen

The IQR fence on order amounts

320

280

350

300

410

290

9800

1/5Seven order amounts in rupees. One looks very different.

The big ideas

What you need to know

IQR rule

Flag values below Q1 - 1.5×IQR or above Q3 + 1.5×IQR. Works well on skewed data.

Z-score rule

Flag values more than about 3 standard deviations from the mean. Best for roughly normal data.

Look at a chart

Box plots and scatter plots show outliers faster than any formula.

Investigate, then act

Fix errors, keep genuine extremes, and always note what you removed and why.

See it in code

Walk through the code

python

1import pandas as pd2 3s = pd.Series([320, 280, 350, 300, 410, 290, 9800])4 5q1, q3 = s.quantile(0.25), s.quantile(0.75)6iqr = q3 - q17low, high = q1 - 1.5 * iqr, q3 + 1.5 * iqr8 9outliers = s[(s < low) | (s > high)]10print(outliers)

1/3Find the first and third quartiles and the gap between them.

The IQR rule in pandas. It flags 9800 and nothing else.

Save yourself the pain

Common mistakes

Oops

Deleting every outlier so the chart looks neat.

Do this instead

Outliers may be fraud, bulk buyers or system bugs, which are often the most interesting finding.

Oops

Using z-scores on heavily skewed data like income.

Do this instead

Use the IQR rule or a log scale. Skew inflates the standard deviation and hides outliers.

Your mission

Hunt for odd values

~25 min

Quick check

You find an order of ₹0 in a sales table. What's the best first step?

Go deeper

1Use a boxplot to visually spot outliers in a sample dataset.
2Learn the IQR (interquartile range) method for flagging outliers systematically.
3Practice deciding, case by case, whether to remove, cap, or investigate an outlier.