Stop 09 · Growth Marketing & ExperimentationTopics · 3/9

A/B Testing for Marketing

In short

An A/B test is a fair race between two versions. Run it badly and you'll 'prove' things that are just luck. Run it well and you get answers you can trust.

Randomly splitting an audience into control and variant groups to measure a change's real causal impact — applied here to landing pages, ads, and emails rather than product features.

The blob flips a shiny coin into the air on a dusty street with a cricket bat nearby.

Think of it like this

The lucky toss that isn't

In gully cricket your friend wins four tosses in a row and declares his coin lucky. It isn't. Small samples throw up streaks by chance. A/B tests need enough visitors before you believe a winner.

Watch it happen

Why peeking early fools you

Day 1: B +40%

Day 2: B +22%

Day 3: B +9%

Day 7: B +3%

Day 14: B +2%

1/4Day 1: B looks 40% better on just 60 visitors. Tempting to stop and celebrate.

The big ideas

What you need to know

Hypothesis first

'If we add reviews near the button, sign-ups will rise, because visitors doubt us.' Write it before testing.

Sample size

Decide how many visitors you need before starting, using a free sample size calculator.

Run full weeks

Weekday and weekend visitors behave differently. Run at least one or two full weeks.

Don't stop on a peek

Stopping the moment one version leads inflates false wins. Stick to the plan.

See it in code

Walk through the code

text

1Hypothesis : Showing "4.6★ from 2,000 students" near the2             button will lift sign-ups, because visitors doubt us.3Metric     : sign-up rate (primary), bounce rate (guardrail)4Variants   : A = current page, B = page + rating line5Split      : 50 / 50, random, all traffic6Sample     : ~6,000 visitors per variant (from calculator)7Duration   : 14 days minimum, no early stopping8Decision   : ship B if sign-ups rise and result is significant

1/4The change and the reason you expect it to work.

A one-page test plan. Writing it before launch stops you from moving the goalposts later.

Save yourself the pain

Common mistakes

Oops

Testing a new button colour on a page with 200 visitors a month.

Do this instead

Low traffic can't detect small changes. Test bigger ideas, or test where traffic is higher.

Oops

Checking results every hour and stopping when B leads.

Do this instead

Set sample size and duration first, then read the result once at the end.

Your mission

Plan a real test

~25 min

Quick check

After one day, version B is ahead by 35% on 80 visitors. What should you do?

Go deeper

1Design an A/B test for a real marketing asset (landing page, ad, or email).
2Learn the difference between correlation and causation, and why A/B tests solve for the latter.
3Understand common pitfalls: stopping a test too early, and testing too many variables at once.