Design & Document an A/B Test
Part of the Product Management Roadmap
In short
This project takes one idea from hunch to a test document a real growth team could run. Doing it once end to end teaches more than reading about ten tests.
The practice project for this phase — take a real hypothesis through the full lifecycle: design, sample size, success metric, guardrails, and a plan for how you'd interpret each possible result.
Think of it like this
The halwa taste test
At a family function, you make two halwas: one with less sugar. Relatives taste both without knowing which is which, and you note every reaction in a notebook. Before starting, you decided how many tasters and what "better" means.
Watch it happen
The build plan for your test doc
1/5Pick a real funnel drop in a public app and gather evidence: reviews, your own use, or friends' feedback.
The big ideas
What you need to know
One-page doc
Problem, hypothesis, variants, metrics, sample size, duration, risks and decision rules.
Show the variants
Include simple mockups of control and variant, even rough Figma frames.
Guardrails
Name what must not get worse, with a threshold that would stop the test.
Pre-commit decisions
Writing decisions before results stops you from moving the goalposts later.
See it in code
Walk through the code
1Test: Show delivery fee on product page2Owner: <your name> Status: Draft3Hypothesis: If..., then +3pt cart completion, because...4Variants: A = current, B = fee shown under price5Primary metric: cart completion rate6Guardrails: avg order value (-2% max), support tickets7Sample: 25,000 users per variant -> 14 days8Decision: ship if primary wins and guardrails hold1/3Name, owner and status, then the full hypothesis.
A starter skeleton. Fill every line before launch; add a results section afterwards.
Save yourself the pain
Common mistakes
Oops
Leaving the decision rule blank until results arrive.
Do this instead
Write it upfront. Otherwise any result can be spun as a win.
Oops
Picking a page with too little traffic to ever reach the sample.
Do this instead
Check traffic first. If the test would take months, test a bolder change or a busier page.
Your mission
First milestone: the draft doc
Quick check
Why write the decision rule before the test runs?
Go deeper
- 1Pick a real feature idea and write a testable hypothesis for it.
- 2Define your primary metric, guardrail metrics, and required sample size.
- 3Write out how you'd interpret a win, a loss, and an inconclusive result before running it.
Stuck on this topic? Ask an Insider
Get 1:1 guidance from people who've walked this exact path — on the InsideEdge app.
