Stop 03 · Data Manipulation & VisualizationTopics · 3/10

Data Cleaning & Wrangling

In short

Real data arrives messy: duplicate rows, typos, blanks and dates stored as text. Cleaning it is often the biggest chunk of any data project, and skipping it quietly ruins your results.

Fixing inconsistent types, formatting, duplicates and outliers before a dataset is usable — the unglamorous work that determines model quality more than any algorithm choice.

The mascot washes vegetables in a big basin and tosses a rotten tomato aside.

Think of it like this

Washing sabzi from the mandi

Vegetables from the market come with mud, a few rotten ones and odd sizes. You wash them, throw out the bad ones and chop everything evenly before cooking. Data cleaning is the same prep work before any analysis.

Watch it happen

Cleaning one 'city' column

Mumbai

mumbai

Bombay

NaN

Delhi

delhi

1/5Six rows, but how many real cities? Spaces, mixed case, an old name and a blank.

The big ideas

What you need to know

Remove duplicates

The same record twice inflates counts. Check df.duplicated().sum() and drop them deliberately.

Fix data types

Numbers stored as text and dates as strings break maths. Convert with to_numeric and to_datetime.

Standardise text

Trim spaces, fix case and map spelling variants so one city doesn't appear as five.

Handle missing values on purpose

Drop, fill with a median, or flag them. Write down which and why, because it changes results.

See it in code

Walk through the code

python

1import pandas as pd2 3df = pd.read_csv("customers.csv")4 5df = df.drop_duplicates()6df["city"] = df["city"].str.strip().str.lower().replace({"bombay": "mumbai"})7df["age"] = pd.to_numeric(df["age"], errors="coerce")8df["age"] = df["age"].fillna(df["age"].median())9df["signup"] = pd.to_datetime(df["signup"], errors="coerce")10 11print(df.isna().sum())

1/4Drop rows that are exact copies of another row.

A typical first cleaning pass. errors='coerce' turns junk like 'twenty' into a blank you can then handle.

Save yourself the pain

Common mistakes

Oops

Cleaning by hand in Excel and forgetting what you changed.

Do this instead

Clean in code. Then the same steps can be rerun on next month's data and checked by others.

Oops

Filling every blank with 0.

Do this instead

A missing age isn't age 0. Use a median, a flag column, or drop it, depending on the column's meaning.

Your mission

Clean a messy file

~35 min

Quick check

An 'amount' column has the dtype object, and df['amount'].sum() joins strings together. What's the fix?

Go deeper

1Take a genuinely messy public dataset and clean it end-to-end.
2Build a checklist: dtypes, duplicates, outliers, inconsistent categories, invalid values.
3Document every cleaning decision so it's reproducible.