A/B Test Analysis Case Study
Part of the Data Analyst Roadmap
In short
Companies run experiments all the time, and someone has to say if the change actually worked. A clear A/B case study shows you can read results honestly, not just run a formula.
Analyze a real or realistic A/B test dataset end-to-end — calculate significance, check guardrails, and write a clear recommendation, directly showcasing Phase 10's skills.
Think of it like this
Two samosa recipes, one stall
A halwai wants to test a new spicy samosa. For a week, every other customer gets the new one. He counts who comes back for a second. Same stall, same days, only the recipe differs. That's an A/B test, and the counting is your job.
Watch it happen
How to analyse a finished test
1/6State it first: "The new checkout button will raise order rate." Name the main metric before looking.
The big ideas
What you need to know
Decide the metric first
Picking the metric after seeing results lets you find a "win" by luck. Write it down before you start.
Check the split
A lopsided split often means a bug in how users were assigned. Results from a broken split can't be trusted.
Lift plus uncertainty
Report the lift and a confidence interval or p-value. "+9%, likely between +1% and +17%" is honest.
Guardrails matter
A change that raises orders but doubles cancellations isn't a win. Always check side effects.
See it in code
Walk through the code
1from statsmodels.stats.proportion import proportions_ztest2 3users = [10_000, 10_050] # group A, group B4orders = [820, 905]5 6print(users[0] / sum(users)) # split check: close to 0.5?7rate_a, rate_b = orders[0] / users[0], orders[1] / users[1]8print(f"A {rate_a:.2%} B {rate_b:.2%} lift {rate_b / rate_a - 1:.1%}")9 10z, p = proportions_ztest(count=orders, nobs=users)11print(f"p-value: {p:.3f}")1/4Users and orders per group. In a real project these come from a SQL query.
A minimal analysis: check the split, compute both rates and the lift, then test if the gap is likely real.
Save yourself the pain
Common mistakes
Oops
Stopping the test the first day B looks better.
Do this instead
Decide the sample size or duration in advance and wait for it. Early peeks often show false wins.
Oops
Reporting only "p < 0.05, B wins".
Do this instead
Report the lift, its range, guardrails and a plain recommendation.
Your mission
First milestone: analyse a sample test
Quick check
Test planned as 50/50, but group A has 12,000 users and B has 8,000. What should you do first?
Go deeper
- 1Find a public A/B test dataset (several exist for practice on Kaggle and elsewhere).
- 2Calculate statistical significance and check for common pitfalls (Phase 10).
- 3Write a recommendation as if presenting to a real product team.
More projects at this stop
End-to-End SQL + BI Dashboard Project
Exploratory Data Analysis (EDA) on a Public Dataset
Business Metrics Deep-Dive Report
Data Cleaning Case Study (Messy Dataset)
Python + Pandas Analysis Notebook
Portfolio Website or GitHub Showcase
Stuck on this topic? Ask an Insider
Get 1:1 guidance from people who've walked this exact path — on the InsideEdge app.
