Stop 10 · A/B Testing & ExperimentationTopics · 8/8

Communicating Experiment Results

In short

A perfect analysis that nobody understands changes nothing. Leaders want three things: what happened, how sure you are, and what to do now. Say those in the first ten seconds.

Presenting a test's outcome clearly to stakeholders who want a simple 'did it work' answer — including the nuance (confidence, guardrails, next steps) without burying the headline.

The jaadu wears headphones and speaks into a microphone beside a cricket bat.

Think of it like this

The radio cricket commentator

Tuning in mid-match, you want the score, who's on top and what's next, before the details of the last over. A good commentator gives that in one breath. Your experiment summary should open the same way, then offer detail for anyone who wants it.

Watch it happen

Sharing a result with the product lead

Analyst

Product lead

Ship it: +0.9pp checkout

1/5Lead with the recommendation and the headline number in plain words.

The big ideas

What you need to know

Answer first

Recommendation and headline number in the first line. Details after, for those who want them.

Ranges, not false precision

'Between +0.2 and +1.6 points' is more honest and more useful than '+0.9 exactly'.

Speak in business terms

Orders, revenue, hours saved. Convert lift into something people feel.

Share failed tests too

A test that didn't win still taught you something. Teams that hide losses repeat them.

See it in code

Walk through the code

text

1TL;DR: Ship "delivery time on cart". Checkout +0.9 pts2(10.0% -> 10.9%), likely between +0.2 and +1.6 pts.3 4Why it matters: about 1,300 extra orders a month at5current traffic.6 7Guardrails: cancellations flat; cart load +40 ms (limit 100).8 9Caveats: 2-week test, outside festive season. Re-check10in December.11 12Next: roll out to 100%, monitor load time for 2 weeks.

1/4Recommendation, headline lift and its range, in the first two lines.

A results note that fits in one Slack message. Numbers here are illustrative; the structure is what to copy.

Save yourself the pain

Common mistakes

Oops

Opening with methodology, p-values and twelve charts.

Do this instead

Open with the decision and headline. Put methods and charts in an appendix.

Oops

Calling a non-significant test 'a failure' or quietly not sharing it.

Do this instead

Share it: what you tested, what you learned, and what you'll try next.

Your mission

The one-message summary

~15 min

Quick check

What should the first line of an experiment summary for a busy leader contain?

Go deeper

1Write a short experiment readout: result, confidence, recommendation, next steps.
2Practice leading with the recommendation, then the supporting evidence.
3Prepare for the common follow-up question: 'are you sure this wasn't just noise?'