Stop 04 · Data & Metrics for PMsTopics · 11/12

Statistical Significance Basics

In short

Your A/B test shows the new button wins by 0.5%. Is it real, or just luck? Statistical significance helps you tell a true win from random noise.

Understanding whether a difference between two numbers (e.g. two A/B test variants) is a real effect or just random noise — enough intuition to avoid shipping a 'winner' that isn't actually one.

The purple blob flips a shiny coin into the air beside a small pile of coins.

Think of it like this

The lucky coin

Your friend flips a coin ten times and gets seven heads. Is the coin rigged? Probably not: that happens by chance quite often. After 1,000 flips with 700 heads, you'd be much more suspicious. More data, more confidence.

Watch it happen

Running a fair A/B test

Hypothesis

Sample size

Run test

p-value

Decide

1/5Hypothesis: "A shorter signup form will raise signup rate." Pick one main metric up front.

The big ideas

What you need to know

Noise is everywhere

Two identical versions will still show different numbers. Significance asks if the gap is bigger than noise.

What a p-value means

A low p-value means the result would be surprising if there were no real effect. It's not the chance you're right.

Sample size first

Small tests can't detect small effects. Decide the sample size before you start.

Significant isn't important

A tiny lift can be statistically real but not worth the effort. Judge the size too.

See it in code

Walk through the code

python

1from statsmodels.stats.proportion import proportions_ztest2 3# Variant B: 156 of 2,400 signed up. Control A: 120 of 2,400.4stat, p_value = proportions_ztest(count=[156, 120], nobs=[2400, 2400])5print(round(p_value, 3))6 7if p_value < 0.05:8    print("Unlikely to be luck alone")9else:10    print("Could be noise, don't ship on this")

1/3statsmodels includes a ready-made test for comparing two conversion rates.

A two-proportion z-test with statsmodels. For these numbers the p-value comes out around 0.03, below the usual 0.05 cut-off.

Save yourself the pain

Common mistakes

Oops

Checking results every day and stopping the moment B looks ahead.

Do this instead

Decide the sample size and duration first, then wait. Peeking makes false wins more likely.

Oops

Reading p = 0.04 as "96% chance B is better".

Do this instead

It means a gap this large would be unusual if there were no real difference. That's all.

Your mission

Test a fake experiment

~30 min

Quick check

Your test is significant, but the new design lifts conversion by only 0.1% and costs months to maintain. What should you do?

Go deeper

1Learn what a p-value intuitively represents, without the full formula derivation.
2Understand why small sample sizes produce unreliable 'wins'.
3Learn the term 'peeking' and why stopping a test early on a good result is a trap.