Stop 09 · Growth & ExperimentationTopics · 4/10

Statistical Significance in Experiments

In short

Variant B is up 4% after two days. Should you ship? Maybe, or maybe it's just noise. Significance helps you tell real effects from lucky streaks.

Applying the stats basics from Phase 4 specifically to A/B tests — knowing your required sample size and test duration before you launch, not deciding to stop once you like the number.

The purple blob tosses a coin into the air with a puzzled, curious face.

Think of it like this

Three heads in a row

You toss a coin three times and get three heads. Is the coin rigged? Probably not. With so few tosses, streaks happen by chance. Toss it 500 times and get 400 heads, and now you'd be suspicious.

Watch it happen

Why peeking early misleads

Day 1: +9%

Day 3: +5%

Day 7: +2%

Day 14: +1.8%

1/5Daily results for variant B versus A, with the test planned for 14 days.

The big ideas

What you need to know

The p-value

Roughly: how surprising your result would be if there were no real difference. Teams often use 0.05 as the cut-off.

Confidence intervals

A range of likely true effects, like +0.5% to +3%. More useful than a single number.

Don't peek and stop

Stopping as soon as it looks significant raises false wins a lot. Fix the duration first.

Significant isn't important

A tiny lift can be statistically real but not worth the cost of shipping.

Save yourself the pain

Common mistakes

Oops

Stopping the test the first day p drops below 0.05.

Do this instead

Decide sample size and duration upfront, and judge at the end.

Oops

Reading p = 0.05 as "95% chance B is better".

Do this instead

It's about how surprising the data is if there's no effect. Use intervals to talk about effect size.

Your mission

Feel randomness in a sheet

~25 min

Quick check

Your test was planned for 2 weeks. On day 2, B is significantly ahead. What should you do?

Go deeper

1Use a free online sample size calculator for a hypothetical A/B test.
2Learn why stopping a test as soon as it looks significant inflates false positives.
3Practice explaining a non-significant result without over-interpreting it as 'no effect'.