Stop 04 · Statistics for AnalystsTopics · 8/11

A/B Testing Statistics

In short

Product teams test everything: button colours, prices, onboarding screens. Analysts make sure the result is real and not a lucky week. Get the stats wrong and you ship changes that do nothing.

Applying hypothesis testing specifically to comparing two variants of a product or feature — the statistical backbone behind Phase 10's experimentation work.

The jaadu at a chai stall pours from a violet kettle and an orange kettle for two customers.

Think of it like this

Two chai recipes at the college fest

At your fest stall, you serve every other customer from the ginger kettle and the rest from the elaichi kettle. You count who comes back for a second cup. Because you alternated fairly, any big gap is probably the recipe, not the customers.

Watch it happen

A clean A/B test

Metric

Sample size

Randomise

Run

Analyse

1/5Pick one primary metric, like checkout conversion, plus a couple of guardrails like refunds.

The big ideas

What you need to know

Plan the sample size

Small lifts need lots of users. Calculate it before starting so you know when to stop.

Don't peek and stop

Stopping the moment p dips below 0.05 inflates false positives. Wait for the planned sample.

Randomise by user

Assign people, not page views, so one person doesn't see both versions.

Watch guardrails

A win on clicks that also raises cancellations or complaints isn't a win.

See it in code

Walk through the code

python

1from statsmodels.stats.power import NormalIndPower2from statsmodels.stats.proportion import proportion_effectsize3 4baseline = 0.040   # current conversion rate5target   = 0.044   # smallest lift worth detecting6 7effect = proportion_effectsize(target, baseline)8n = NormalIndPower().solve_power(effect, alpha=0.05, power=0.8)9print(round(n), "users per group")

1/3Your current rate and the smallest improvement that matters to the business.

Sample size for detecting 4.0% to 4.4% with 80% power. It comes out at roughly 20,000 users per group.

Save yourself the pain

Common mistakes

Oops

Checking results every morning and stopping at the first 'significant' day.

Do this instead

Fix the sample size or duration upfront. If you must monitor early, use methods designed for it.

Oops

Testing ten metrics and celebrating whichever one moved.

Do this instead

Choose one primary metric in advance. Treat the rest as clues for future tests.

Your mission

Plan a test

~25 min

Quick check

After 2 days of a planned 14-day test, B is ahead with p = 0.04. What should you do?

Go deeper

1Walk through calculating statistical significance for a simple two-group comparison.
2Learn why sample size and effect size both matter for a trustworthy result.
3Preview Phase 10 to see where this statistical grounding gets applied.