Stop 10 · A/B Testing & ExperimentationTopics · 1/8

Designing an A/B Test

In short

'The new checkout button will increase orders.' Says who? An A/B test settles the argument with evidence instead of whoever is most senior in the meeting.

Defining the change, the hypothesis, the metric, and the split before launching — a well-designed test answers a clean question; a poorly designed one produces an ambiguous result nobody can act on.

The jaadu hands out samosa plates with two different chutneys to students in a line.

Think of it like this

Two samosa recipes in the canteen

The canteen uncle wants to try a new chutney. On random tables he serves the new one, on others the old one, same samosas, same week. Then he counts which plates come back empty. Random assignment is what makes the comparison fair.

Watch it happen

From idea to decision

Hypothesis

Metric

Randomise

Run

Decide

1/5Write it down: 'Showing delivery time on the cart page will raise checkout rate, because users worry about delays.'

The big ideas

What you need to know

A clear hypothesis

Change, expected effect, and why. 'Let's see what happens' is not a hypothesis.

One primary metric

Decide the success metric before launch. Choosing afterwards lets you find a 'win' in noise.

Guardrail metrics

Things that must not get worse, like cancellations, support tickets or app crashes.

Randomise the right unit

Usually users, not page views. Otherwise one person sees both versions and muddies the result.

See it in code

Walk through the code

text

1TEST PLAN: Delivery time on cart page2Hypothesis: Showing estimated delivery time raises checkout3            rate because users worry about delays.4Unit:       Logged-in user, 50/50 split5Primary:    Checkout rate (checkouts / cart views)6Guardrails: Cancellation rate, cart page load time7MDE:        +1 percentage point (10% -> 11%)8Sample:     ~14,700 users per group9Duration:   2 full weeks10Decision:   Ship if primary is significant and positive11            and no guardrail worsens significantly.

1/4The change and why you expect it to work.

A one-page test plan, agreed before launch. It stops arguments later about what 'success' meant.

Save yourself the pain

Common mistakes

Oops

Testing five changes at once in one variant.

Do this instead

If it wins, you won't know which change did it. Test one idea per variant, or design a multivariate test.

Oops

Deciding the success metric after seeing the results.

Do this instead

Write the primary metric and decision rule in the plan before launch.

Your mission

Write a test plan

~20 min

Quick check

Why randomise users into A and B instead of letting them choose?

Go deeper

1Design a full A/B test plan for a real feature idea: hypothesis, metric, and split.
2Learn to define your primary metric and 1-2 guardrail metrics upfront.
3Practice stating what result would make you call the test a win, loss, or inconclusive.