Stop 07 · Usability & Heuristic EvaluationTopics · 6/8

A/B Testing for Design

In short

Two designs, both look good, and the team can't agree. An A/B test settles it with real behaviour: show each version to a random half of users and measure which one works better.

Testing two design variants against real users to measure which performs better on a specific metric — applying statistical rigor to design decisions rather than relying purely on opinion.

The jaadu runs a tea stall with two different kettles, pouring cups for a line of customers.

Think of it like this

Two chai recipes at the stall

The chaiwala makes adrak chai on Monday, Wednesday and Friday mornings and elaichi chai on the others, then counts cups sold. Same stall, same crowd, one change. After a few weeks he knows which one people really prefer.

Watch it happen

How one A/B test works

Visitors

Splitter

Version A

Version B

Open product page

1/5Every visitor arrives at the same URL. They don't know a test is running.

The big ideas

What you need to know

One goal, decided first

Pick the metric before the test starts, like checkout completion. Choosing after you see data is cheating yourself.

Change one thing

If B changes the colour, text and layout together, you won't know which change made the difference.

You need enough people

Small samples give random results. Use a sample size calculator and wait until you reach it.

It tells what, not why

A/B tests show which version wins. Pair them with user interviews to understand the reason.

See it in code

Walk through the code

excel

1|   | Visitors | Orders | Rate      |2| A | 4800     | 192    | =C2/B2    |3| B | 4750     | 228    | =C3/B3    |4 5Lift: =(D3-D2)/D2

1/2Each version's conversion rate: orders divided by visitors. Here A is 4% and B is 4.8%.

Conversion rate is orders divided by visitors. Lift shows how much better B did. Check significance with a proper calculator before deciding.

Save yourself the pain

Common mistakes

Oops

Stopping the test the moment B looks ahead.

Do this instead

Decide the sample size in advance and wait for it. Early peeks often show random swings.

Oops

A/B testing tiny pages with a few dozen visitors a day.

Do this instead

Low traffic won't give clear answers. Use usability testing there instead.

Your mission

Plan an A/B test

~25 min

Quick check

Version B changed the headline, button colour and image all at once, and it won. What's the problem?

Go deeper

1Design 2 variants of the same screen testing one specific hypothesis.
2Learn why testing only one variable at a time gives you a clean, trustworthy result.
3Understand A/B testing answers 'which performs better', not 'why' — pair it with qualitative research.