Stop 09 · Growth & ExperimentationTopics · 6/10

Experiment Design & Sample Size

In short

Run a test on too few users and you'll miss real wins or celebrate fake ones. Working out sample size before you start is what makes the result worth trusting.

Deciding what to measure, how long to run a test, and how many users you need before you can trust the result — under-designed experiments produce confident-sounding, wrong conclusions.

The purple blob tastes biryani with a long spoon from a huge cooking pot.

Think of it like this

Tasting the biryani

To check a huge pot of biryani, one spoon from the top isn't enough. The masala may sit lower. You taste from a few spots. The bigger the pot and the subtler the flavour you're checking, the more tastes you need.

Watch it happen

Smaller effects need more users

Lift 20%: few k

Lift 10%: more

Lift 5%: lots

Lift 2%: huge

1/5Sample size depends on your baseline rate, the smallest lift you care about, and how sure you want to be.

The big ideas

What you need to know

Minimum detectable effect

The smallest change worth knowing about. Smaller means more users and longer tests.

Use a calculator

Free online sample size calculators take baseline, effect, significance and power.

Power

The chance of catching a real effect if it exists. 80% is a common choice.

Duration from traffic

Divide the needed sample by daily eligible users, then round up to full weeks.

See it in code

Walk through the code

excel

1A1: Needed users per variant   B1: 250002A2: Variants                   B2: 23A3: Eligible users per day     B3: 40004A4: Days needed                B4: =ROUNDUP(B1*B2/B3, 0)5A5: Weeks (round up)           B5: =ROUNDUP(B4/7, 0)

1/3Take the per-variant number from a sample size calculator, times the number of variants.

Turn a calculator's sample size into a test duration. Here it's 13 days, so you'd run two full weeks.

Save yourself the pain

Common mistakes

Oops

Starting a test and "seeing how it goes".

Do this instead

Calculate sample size and duration before launch, and write them in the test doc.

Oops

Testing tiny changes on a low-traffic page.

Do this instead

Test bolder changes, or pick a page with enough traffic to detect the effect.

Your mission

Size a real test

~25 min

Quick check

You want to detect a smaller lift than before, with everything else the same. What happens to the sample size?

Go deeper

1Practice calculating (or estimating) required sample size for a small hypothetical test.
2Learn why low-traffic features are hard to A/B test reliably.
3Understand the trade-off between test duration and time-to-decision.