Stop 06 · Python for Data AnalysisTopics · 5/12

Data Cleaning With Pandas

In short

You cleaned data in Excel by hand. Next month the same messy file arrives and you start again. In Pandas you write the cleaning once and rerun it forever.

Using Pandas to handle missing values, duplicates, and inconsistent types programmatically — the same cleaning tasks from Excel (Phase 2) and Phase 8, but faster and repeatable via code.

The jaadu picks small stones out of a steel plate of rice.

Think of it like this

Picking stones from rice

Before cooking, someone at home spreads rice on a plate and picks out stones and husk. Skip it and every bite is a gamble. Cleaning data is that same patient sorting, but in Pandas you teach the code which bits to pick out.

Watch it happen

A repeatable cleaning pass

Raw file

Fix text

Fix types

Dedupe

Missing

Check

1/61,200 rows arrive. City says 'mumbai ', 'Mumbai' and 'MUMBAI'. Amount has '₹1,200'.

The big ideas

What you need to know

Find missing values

df.isna().sum() counts blanks per column. Decide drop or fill column by column, not all at once.

Duplicates

drop_duplicates() removes repeated rows. Use subset= to define what counts as the same record.

.str for text

.str.strip(), .str.title() and .str.replace() clean whole text columns in one go.

Wrap it in a function

Put all steps in def clean(df): and next month's file is one line of work.

See it in code

Walk through the code

python

1def clean(df):2    df = df.copy()3    df["city"] = df["city"].str.strip().str.title()4    df["amount"] = (5        df["amount"].astype(str)6        .str.replace("₹", "").str.replace(",", "")7        .astype(float)8    )9    df = df.drop_duplicates()10    df = df.dropna(subset=["order_id"])11    df["coupon"] = df["coupon"].fillna("none")12    return df13 14before = len(raw); orders = clean(raw)15print(before, "->", len(orders))

1/4Work on a copy so the raw data stays untouched for comparison.

A cleaning function you can reuse on every new export. Always print row counts so nothing disappears silently.

Save yourself the pain

Common mistakes

Oops

Calling df.dropna() on the whole table and losing half the rows.

Do this instead

Check isna().sum() first and drop only on the columns that truly must be present.

Oops

Cleaning by editing the CSV by hand, then writing code on the edited file.

Do this instead

Keep the raw file untouched. All fixes live in code so anyone can repeat them.

Your mission

Write your clean() function

~35 min

Quick check

Your amount column has values like '₹1,250' and info() shows dtype object. What must happen before you can sum it?

Go deeper

1Practice .isnull(), .dropna(), and .fillna() on a dataset with missing values.
2Use .drop_duplicates() and verify the row count before and after.
3Write a reusable cleaning function you could apply to future similar datasets.