Stop 10 · A/B Testing & ExperimentationTopics · 6/8

Reading an Experiment Readout

In short

Experiment platforms spit out a wall of green and red numbers. Knowing what to read first, and what to ignore, stops you from shipping a broken test or killing a good one.

Interpreting a finished test's results — the effect size, confidence interval, and guardrail metrics together — rather than just checking if the headline p-value crossed 0.05.

The jaadu reads a report on a clipboard in a clinic chair.

Think of it like this

Reading a blood test report

A lab report lists dozens of values with normal ranges. You don't panic at the first bold number. You check it's your report, look at the values your doctor asked about, then the flagged ones. An experiment readout needs that same calm reading order.

Watch it happen

Read a readout in this order

Split OK?

Primary

Interval

Guardrails

Segments

Decision

1/6Sanity first: is the split 50/50, did it run the planned time, any tracking alerts?

The big ideas

What you need to know

Sanity before success

If the split is off or tracking broke, every other number is suspect. Check first.

Primary metric decides

The other twenty metrics are context. Don't let a green secondary metric override the plan.

Relative vs absolute lift

10% to 11% is +1 point absolute and +10% relative. Always say which one you mean.

Guardrails can veto

A win on checkouts with a jump in cancellations might not be a win at all.

Save yourself the pain

Common mistakes

Oops

Jumping to whichever metric is greenest.

Do this instead

Follow the order: sanity, primary, interval, guardrails, then segments.

Oops

Saying '+10%' when you mean 10% to 11%.

Do this instead

Say '+1 percentage point, a 10% relative lift'. Mixing them confuses everyone.

Your mission

Read a mock readout

~20 min

Quick check

Conversion moved from 4% to 5%. Which statement is accurate?

Go deeper

1Practice reading a sample experiment readout and summarizing it in 2 sentences.
2Check the guardrail metrics, not just the primary metric, before calling a test a win.
3Learn to flag a result that's statistically significant but practically too small to matter.