A/B Testing & Experimentation Questions
Part of the Data Science Roadmap
In short
'How would you know if the new model actually helps?' The honest answer is usually an A/B test. Interviewers check that you can design one properly and read the result without fooling yourself.
'How would you test whether this new model actually improves outcomes?' — connecting Phase 12's deployment A/B testing with the underlying statistics from Phase 2.
Think of it like this
The chai-wala's experiment
A chai-wala wants to know if adding elaichi sells more. Trying it only on Sunday proves nothing, the crowd is different anyway. So he alternates: one customer gets recipe A, the next gets B, for two weeks, and counts repeat orders. Random split, fixed plan, then compare.
Watch it happen
Running a fair A/B test
1/6State it clearly: 'The new ranking model increases orders per user.'
The big ideas
What you need to know
Randomise the right unit
Usually users. If friends influence each other, you may need to randomise by city or group.
Size it first
Small tests miss real effects. Decide sample size and duration before you start.
Don't peek and stop
Stopping the moment p < 0.05 inflates false positives. Stick to the plan or use sequential methods.
Significant isn't always useful
A tiny lift can be statistically real but not worth the cost. Discuss both.
See it in code
Walk through the code
1from statsmodels.stats.proportion import proportions_ztest2 3conversions = [530, 584] # control, treatment4visitors = [10000, 10000]5 6stat, p_value = proportions_ztest(conversions, visitors)7lift = conversions[1] / visitors[1] - conversions[0] / visitors[0]8 9print(f"Absolute lift: {lift:.2%}, p-value: {p_value:.3f}")1/3Conversions and visitors for control (A) and treatment (B).
A two-proportion z-test. With these numbers the p-value is around 0.1, so the lift could still be noise.
Save yourself the pain
Common mistakes
Oops
Checking every day and stopping on the first day the result looks significant.
Do this instead
Fix the sample size and duration up front, then analyse once at the end.
Oops
Randomising by page view, so one user sees both versions.
Do this instead
Randomise by user ID so each person has one consistent experience.
Your mission
Plan a test on paper
Quick check
You check an A/B test daily and stop on the first day p < 0.05. What's wrong?
Go deeper
- 1Review Phase 12's A/B testing content and practice designing a test for a model rollout.
- 2Prepare to discuss sample size, guardrail metrics, and how you'd interpret each possible result.
- 3Practice explaining why offline model metrics don't guarantee real-world business impact.
More case studies & product sense at this stop
ML System Design Questions
Case Study / Business Problem Questions
Stuck on this topic? Ask an Insider
Get 1:1 guidance from people who've walked this exact path — on the InsideEdge app.
