Stop 08 · Unsupervised LearningTopics · 5/7

Anomaly Detection

In short

Fraud, machine failures and hacked accounts are rare, and you often have no labels for them. Anomaly detection learns what 'normal' looks like and flags what doesn't fit.

Identifying data points that don't fit the normal pattern — used for fraud detection, equipment failure prediction, and data quality monitoring, often building on clustering or density-based methods.

The mascot holds a magnifying glass over one odd blue tomato in a crate of red ones.

Think of it like this

The odd tomato in the crate

A sabziwala sorts a crate of tomatoes every morning. He doesn't need a list of every way a tomato can go bad. He knows what a good tomato looks like, so the one that's oddly coloured or squishy jumps out straight away.

Watch it happen

A week of UPI spends (in rupees)

480

520

450

610

25000

500

530

1/4Daily spends for one user. Most days sit around 450 to 600 rupees.

The big ideas

What you need to know

Learn normal, flag the rest

Most methods model typical behaviour and score how far each point is from it.

Isolation Forest

Random splits isolate odd points in very few cuts. Fewer cuts needed means more anomalous.

Simple rules still work

Z-scores or the IQR rule catch outliers in one column fast. Start there before anything fancy.

Humans confirm

Flags need review. Feedback from reviewers can later become labels for a supervised model.

See it in code

Walk through the code

python

1import numpy as np2from sklearn.ensemble import IsolationForest3 4rng = np.random.default_rng(0)5normal = rng.normal(loc=500, scale=80, size=(200, 1))   # daily UPI spends6odd = np.array([[5000], [7200], [20]])7X = np.vstack([normal, odd])8 9iso = IsolationForest(contamination=0.02, random_state=0).fit(X)10flags = iso.predict(X)          # -1 = anomaly, 1 = normal11print(X[flags == -1].ravel())

1/3200 normal days plus three strange ones, stacked into one array.

Isolation Forest flags the points that are easiest to cut away from the rest. The big spends should be among them.

Save yourself the pain

Common mistakes

Oops

Setting contamination high so more points get flagged 'just in case'.

Do this instead

Base it on what reviewers can handle and what you know about the data. Too many flags get ignored.

Oops

Treating every outlier as an error and deleting it.

Do this instead

Investigate first. Some outliers are the most valuable rows, like a real fraud or a new trend.

Your mission

Find odd days in a public dataset

~30 min

Quick check

Why is Isolation Forest good at spotting anomalies?

Go deeper

1Try a simple anomaly detection method (e.g. Isolation Forest) on a sample dataset.
2Learn why anomaly detection is often framed as an unsupervised problem (labeled anomalies are rare).
3Understand the trade-off between catching more anomalies and generating more false alarms.