Stop 17 · MLOps & Model DeploymentTopics · 7/10

A/B Testing Models

In short

Your new model scores better offline. Will real users actually click more, buy more or complain less? An A/B test answers that with live traffic instead of hope.

Running two model versions simultaneously on live traffic to compare real-world performance before fully committing to a rollout.

The blob runs a chai stall pouring from two different kettles into two rows of cups.

Think of it like this

The chai stall's new recipe

The chaiwala isn't sure about adding elaichi, so he serves the new recipe to one in ten customers and watches who comes back for a second cup. If they do, everyone gets the new chai. That's an A/B test: a small slice tries the new model first.

Watch it happen

Routing users between two models

User u103

Router

Model A

Model B

GET /recommendations

1/5A user asks for recommendations. The router decides which model serves them.

The big ideas

What you need to know

Sticky buckets

Hash the user id so each user always sees the same model. Random per request makes results noisy and confusing.

Pick one metric first

Decide the success metric before starting, like click rate or resolved tickets. Don't go hunting after.

Enough users, enough time

Small differences need lots of traffic to be real. Use a sample size calculator and don't stop early.

Watch guardrail metrics

Also track latency, errors and complaints. A model that wins clicks but doubles latency may still lose.

See it in code

Walk through the code

python

1import hashlib2 3def bucket(user_id: str, treatment_share: float = 0.1) -> str:4    h = int(hashlib.sha256(user_id.encode()).hexdigest(), 16) % 1005    return "model_b" if h < treatment_share * 100 else "model_a"6 7for uid in ["u101", "u102", "u103", "u104"]:8    print(uid, bucket(uid))

1/3Hash the id into a stable number from 0 to 99. sha256 gives the same result on every machine.

Deterministic bucketing. The same user id always lands in the same bucket, on every server.

Save yourself the pain

Common mistakes

Oops

Using Python's built-in hash() for buckets.

Do this instead

hash() on strings changes between processes. Use hashlib so every server agrees.

Oops

Stopping the test the first day B looks ahead.

Do this instead

Early leads are often noise. Decide the test length up front and wait for it.

Your mission

Simulate an A/B test

~30 min

Quick check

Why hash the user id instead of picking a random model on each request?

Go deeper

1Design an A/B test comparing two model versions on a clear success metric.
2Learn how to split traffic safely and avoid contaminating your comparison.
3Understand statistical significance basics before declaring a 'winner.'