Stop 10 · A/B Testing & ExperimentationTopics · 3/8

Statistical Significance in A/B Tests

In short

B converted at 10.8% and A at 10%. Is B better, or just lucky this week? Significance testing asks how surprising that gap would be if nothing had really changed.

Applying Phase 4's hypothesis testing directly to a live experiment — checking whether an observed difference between variants is likely real, not noise, before rolling out a change.

The jaadu flips a coin while a friend watches with crossed arms.

Think of it like this

The 'lucky' coin

Your friend flips a coin ten times, gets six heads, and calls it lucky. You shrug: six out of ten happens all the time with a normal coin. Sixty out of a hundred would be more convincing. Significance is asking how often plain luck would produce what you saw.

Watch it happen

Same rates, different sample sizes

A: 1,000/10,000

B: 1,080/10,000

1/5A converts 10.0%, B converts 10.8%, with 10,000 users each.

The big ideas

What you need to know

What a p-value means

If there were truly no difference, how often would you see a gap this big or bigger? Small p means rarely.

What it doesn't mean

p = 0.03 is not '97% chance B is better'. It's a statement about the data, assuming no effect.

Confidence intervals

A range of plausible true lifts. More useful than a yes/no, because it shows the size too.

Significant isn't always important

With huge samples, a 0.01% lift can be significant and still not worth the engineering cost.

See it in code

Walk through the code

python

1from statsmodels.stats.proportion import proportions_ztest, confint_proportions_2indep2 3conv = [1080, 1000]      # B, A4users = [10000, 10000]5 6z, p = proportions_ztest(conv, users)7low, high = confint_proportions_2indep(conv[0], users[0], conv[1], users[1])8 9print(f"lift: {conv[0]/users[0] - conv[1]/users[1]:.2%}")10print(f"p-value: {p:.3f}")11print(f"95% CI for lift: {low:.2%} to {high:.2%}")

1/4Conversions and users for each group, B first.

Prints a lift of 0.80%, a p-value near 0.06 and an interval that crosses zero. Not enough evidence yet.

Save yourself the pain

Common mistakes

Oops

Saying 'p = 0.06, so there's no effect.'

Do this instead

Say the test didn't find enough evidence. The interval shows a real effect is still possible.

Oops

Reporting only 'B won, p < 0.05.'

Do this instead

Report the lift and its confidence interval so people see how big the gain might be.

Your mission

Run the numbers

~20 min

Quick check

A test gives p = 0.02 and a lift of +0.05% on a huge sample. What's the best reading?

Go deeper

1Calculate significance for a simple two-variant test using real or sample numbers.
2Learn why 'peeking' at results early and stopping on a good day inflates false positives.
3Practice explaining a non-significant result honestly, without over-interpreting it.