Stop 10 · A/B Testing & ExperimentationTopics · 4/8

Common Experimentation Pitfalls

In short

Most failed A/B tests don't fail on maths. They fail on process: stopping early, uneven groups, or checking twenty metrics until one looks good. Knowing the traps keeps your results honest.

Novelty effects (a change performs well just because it's new), seasonality, sample ratio mismatch, and running too many tests at once without correction — the recurring ways experiments mislead teams.

The jaadu opens an oven door early and sees a sunken cake.

Think of it like this

Opening the oven too early

Open the oven every few minutes to check the cake and it sinks. You've changed the result by peeking. Checking an A/B test daily and stopping the moment it looks good does the same thing: you end up baking in a result that isn't really there.

Watch it happen

Where tests go wrong

A/B test

Before

During

After

Too few users

Peeking

Uneven split

Many metrics

Novelty effect

1/5Underpowered: too few users to detect a realistic effect. A 'no result' tells you nothing.

The big ideas

What you need to know

Don't peek and stop

Fix the sample size in advance, or use a method designed for continuous monitoring.

Check the split

A chi-square test on group sizes catches broken assignment before you trust any metric.

Pre-register metrics

Primary and guardrails chosen before launch. Everything else is exploratory, labelled as such.

Watch for spillover

If A and B users interact, like riders and drivers sharing a market, one group affects the other.

See it in code

Walk through the code

python

1from scipy.stats import chisquare2 3a_users, b_users = 50_812, 48_9304total = a_users + b_users5expected = [total / 2, total / 2]6 7stat, p = chisquare([a_users, b_users], f_exp=expected)8print(f"split: {a_users/total:.1%} / {b_users/total:.1%}, p = {p:.2g}")9 10if p < 0.001:11    print("Sample ratio mismatch: investigate before reading results")

1/3Users actually in each group, and what a perfect 50/50 split would be.

A sample ratio mismatch check. This 50.9/49.1 split looks small, but with this many users it's very unlikely by chance.

Save yourself the pain

Common mistakes

Oops

Stopping the test on day 3 because B is 'significant'.

Do this instead

Run to the planned sample size and full weeks. Early significance is often noise.

Oops

Ignoring a 53/47 split because 'it's close enough'.

Do this instead

Run an SRM check. An uneven split often means one group lost users due to a bug.

Your mission

See peeking in action

~30 min

Quick check

A 50/50 test ends with 51,000 users in A and 47,000 in B. What should you do first?

Go deeper

1Learn the term 'novelty effect' and how a longer test duration helps detect it.
2Read about sample ratio mismatch (SRM) and why it can silently invalidate a test.
3List the pitfalls most likely to show up in your own team's testing process.