Stop 08 · Data Cleaning & WranglingTopics · 1/9

Identifying Data Quality Issues

In short

Bad data doesn't announce itself. It looks like a normal table and quietly gives you wrong answers. Spotting problems before you analyse is what makes people trust your numbers.

The first, often-skipped step — profiling a new dataset for missing values, inconsistent formats, duplicates and impossible values before trusting a single number from it.

The jaadu inspects tomatoes at a vegetable cart with a magnifying glass.

Think of it like this

Checking tomatoes at the mandi

At the vegetable cart you don't just grab a kilo. You check for bruises, squashed ones and a stone hidden in the weighing pan. A quick inspection before buying saves a bad dinner. Data needs the same quick check before you cook anything with it.

Watch it happen

Five things to check

Data quality

Complete?

Valid?

Unique?

Consistent?

Fresh?

1/5Completeness: are required fields filled? 12% of orders have no city.

The big ideas

What you need to know

Profile every new dataset

Row count, nulls per column, unique counts, min and max. Ten minutes that prevents days of rework.

Plot the distributions

A histogram shows impossible values and weird spikes much faster than scrolling rows.

Reconcile with a known total

Does your revenue match finance's number for last month? If not, find out why before going further.

Ask the data owner

Some oddities are real, like a sale day. Ask before you 'fix' something that wasn't broken.

See it in code

Walk through the code

python

1print(df.shape)2print(df.isna().mean().round(3).sort_values(ascending=False))3print(df.nunique())4print(df.describe(include="all").T)5 6print("dup order ids:", df["order_id"].duplicated().sum())7print("future dates:", (df["order_date"] > pd.Timestamp.today()).sum())8print("negative amounts:", (df["amount"] < 0).sum())9print("latest row:", df["order_date"].max())

1/4Size, then the share of missing values per column, worst first.

A quick profile you can paste into any notebook. Each line answers one of the five quality questions.

Save yourself the pain

Common mistakes

Oops

Jumping straight into charts with a dataset you've never profiled.

Do this instead

Profile first, write down the issues you find, then decide how to handle each one.

Oops

Silently fixing oddities without telling anyone.

Do this instead

Report issues to the data owner too. Fixing them upstream helps everyone who uses the data.

Your mission

Quality report card

~25 min

Quick check

Your customer table has 'Bengaluru', 'Bangalore' and 'BLR' as separate cities. Which quality issue is this?

Go deeper

1Run a basic profile (row counts, null counts, unique value counts) on any dataset before analyzing it.
2Build a personal checklist of the top 5 issues you always check for first.
3Practice on a deliberately messy public dataset to build pattern recognition.