Stop 09 · Growth & ExperimentationTopics · 2/10

A/B Testing Fundamentals

In short

Opinions are cheap and often wrong. An A/B test shows whether a change actually caused a difference, because the only thing different between groups is your change.

Randomly splitting users into a control and variant group to measure a change's real causal impact — the gold standard for validating that a feature actually improved the metric you care about.

The purple blob at a chai stall hands cups from two different kettles to two customers.

Think of it like this

Two kettles at the chai stall

The chaiwala wonders if more ginger sells better. He serves every other customer from the new kettle and counts who orders a second cup. Same stall, same day, same crowd. Only the recipe differs, so the result is fair.

Watch it happen

How a user lands in a test

User

Assigner

Analytics

opens checkout

1/5A user enters the part of the app being tested.

The big ideas

What you need to know

Random assignment

Randomness makes groups alike on average, so differences come from your change.

One primary metric

Decide upfront the one metric that decides the test, like checkout conversion.

Guardrail metrics

Also watch things that must not get worse, like refunds or page load time.

Run full weeks

Behaviour differs by weekday. Many teams run at least one or two full weeks.

Save yourself the pain

Common mistakes

Oops

Changing five things at once and calling it one test.

Do this instead

If you change many things, you'll know it worked but not why. Test focused changes.

Oops

Comparing this month's new design with last month's numbers.

Do this instead

Run both versions at the same time. Seasons, sales and news distort before-and-after comparisons.

Your mission

Design an A/B test on paper

~25 min

Quick check

Why compare A and B at the same time instead of before and after?

Go deeper

1Design an A/B test for a real feature idea, including control and variant.
2Learn the difference between correlation and causation, and why A/B tests solve for the latter.
3Understand common pitfalls: novelty effects, seasonality, and running tests too briefly.