Stop 08 · Data Cleaning & WranglingTopics · 6/9

Standardizing & Normalizing Data

In short

'Bangalore' vs 'Bengaluru' splits your city chart in two. Marks out of 50 vs out of 100 make toppers look average. Standardising and normalising put things on the same footing.

Making values comparable — consistent units, consistent categorical labels ('NY' vs 'New York' vs 'ny'), and, for some statistical methods, rescaling numeric ranges.

The jaadu cuts a small and a large pizza into equal numbers of slices.

Think of it like this

Slicing two pizzas fairly

A small pizza and a large one can't be compared slice by slice if you cut them differently. Cut both into eight and 'three slices' means the same share of each. Normalising rescales numbers so comparisons are fair. Standardising makes labels follow one format.

Watch it happen

Same label, then same scale

Bangalore

bengaluru

BLR

Bengaluru

1/5Four rows, one city, written four ways. A group-by would show four cities.

The big ideas

What you need to know

Standardise text

One spelling, one case, one format for phones and dates. Keep a mapping table for known variants.

Units

Convert grams and kilograms, paise and rupees, to one unit before adding or comparing.

Min-max scaling

Squeezes values into 0 to 1. Easy to explain, but one extreme value squashes everyone else.

Z-score

(x - mean) / std tells how many standard deviations from average. Useful when comparing different measures.

See it in code

Walk through the code

python

1city_map = {"bangalore": "Bengaluru", "bengaluru": "Bengaluru",2            "blr": "Bengaluru", "bombay": "Mumbai", "mumbai": "Mumbai"}3key = df["city"].str.strip().str.lower()4df["city"] = key.map(city_map).fillna(df["city"].str.strip().str.title())5 6df["weight_kg"] = df["weight"].where(df["unit"] == "kg", df["weight"] / 1000)7 8s = df["score"]9df["score_minmax"] = (s - s.min()) / (s.max() - s.min())10df["score_z"] = (s - s.mean()) / s.std()

1/4A small mapping table of known variants. Keep it in one place so it's easy to extend.

Map city variants to one name, convert grams to kilograms, and scale scores two ways.

Save yourself the pain

Common mistakes

Oops

Fixing city spellings by hand in Excel each month.

Do this instead

Keep a mapping table in code or a sheet. New variants get added once and fixed forever.

Oops

Adding up a weight column where some rows are grams and some kilograms.

Do this instead

Check for a unit column or suspicious ranges, and convert everything to one unit first.

Your mission

One city, one name

~20 min

Quick check

Scores are 10, 20 and 50. What does min-max scaling give for 20?

Go deeper

1Standardize a messy categorical column (inconsistent casing/spelling) into clean values.
2Learn the difference between normalization (0-1 scale) and standardization (z-scores).
3Practice building a mapping table for messy categorical values to canonical ones.