In short
One customer counted twice inflates your user base. One order loaded twice inflates revenue. Duplicates are sneaky because each copy looks perfectly normal on its own.
Distinguishing true duplicate records from legitimately repeated events (e.g. the same customer ordering twice) before blindly deduplicating a dataset.
Think of it like this
Two invites for one uncle
Planning a wedding, you merge your list with your cousin's. Sharma uncle is on both, once as 'R. Sharma' and once as 'Rajesh Sharma'. Send both and he gets two cards and you print an extra. Spotting him needs more than an exact match.
Watch it happen
Exact vs near duplicates
1/4Five sign-ups by email. drop_duplicates() as is would remove only one row, the exact repeat.
The big ideas
What you need to know
Exact duplicates
Every column identical, often from a file loaded twice. drop_duplicates() handles these.
Key duplicates
Same ID, different details. Decide which version wins: latest update, most complete, or a rule from the business.
Normalise before matching
Lowercase, trim spaces, standard phone formats. Many 'different' values are really the same.
keep= decides who stays
Sort first, then keep="first" or "last". Sorting by updated time keeps the newest record.
See it in code
Walk through the code
1print("exact dups:", df.duplicated().sum())2df = df.drop_duplicates()3 4df["email_key"] = df["email"].str.strip().str.lower()5print("same email:", df["email_key"].duplicated().sum())6 7df = (8 df.sort_values("updated_at")9 .drop_duplicates(subset="email_key", keep="last")10)11print(len(df), "unique customers")1/3Count and remove rows where every column is identical.
Remove exact copies, then match on a cleaned key and keep the most recently updated record.
Save yourself the pain
Common mistakes
Oops
Removing rows that look the same but are genuinely separate events.
Do this instead
Two identical ₹50 chai orders on one day can be real. Define a duplicate using an ID or the business rule.
Oops
Deduplicating without recording how many rows were removed.
Do this instead
Log counts before and after. A sudden jump in duplicates often means a pipeline bug.
Your mission
Find the hidden twins
Quick check
Customers appear several times with the same ID but different addresses over time. What's the usual fix?
Go deeper
- 1Practice identifying duplicates using a full-row match versus a key-column match.
- 2Investigate a few 'duplicate' rows manually before deciding whether to remove them.
- 3Learn why deduplication logic should be documented, not just applied silently.
More topics at this stop
Identifying Data Quality Issues
Handling Missing Data
Data Type Conversion
Outlier Handling
Standardizing & Normalizing Data
Data Validation Rules
ETL Basics (Extract, Transform, Load)
Documenting Data Cleaning Decisions
Stuck on this topic? Ask an Insider
Get 1:1 guidance from people who've walked this exact path — on the InsideEdge app.
