Stop 04 · Statistics for AnalystsTopics · 4/11

Sampling & Sampling Bias

In short

You rarely get data on everyone. You get a sample. If the sample is skewed, your answer is wrong, however clever your analysis, and nothing in the numbers will warn you.

Drawing conclusions about a full population from a smaller sample — and the many ways a sample can be unrepresentative (selection bias, survivorship bias) and quietly mislead you.

The jaadu tastes a spoonful from the top of a big pot of layered biryani.

Think of it like this

Tasting only the top of the biryani

If you taste one spoon from the top of a dum biryani, you'll say it's plain rice. The masala and meat are at the bottom. A good taste needs a spoon that reaches all layers. A good sample needs to reach all kinds of people.

Watch it happen

Two ways to survey app users

Pop-up in the app

Only reaches active users
Unhappy users who left never see it
Satisfaction looks higher than reality

Random sample of all users

Includes people who stopped using it
Every user has a fair chance
Results reflect the whole base

Ask who could not end up in your sample. That missing group is usually where the bias hides.

The big ideas

What you need to know

Random means fair chance

In a random sample every member of the population has a known, fair chance of being picked.

Selection bias

Your collection method favours some people, like surveying only on weekday mornings.

Survivorship bias

Looking only at those who made it. Studying only successful startups hides why others failed.

Self-selection bias

People who choose to respond, like reviewers, are often the very happy or very angry.

Save yourself the pain

Common mistakes

Oops

Thinking a huge sample fixes bias.

Do this instead

A million biased responses are still biased. Fix how you collect before you worry about size.

Oops

Generalising from one city or one week to the whole country or year.

Do this instead

State the limits: 'This is for Mumbai users in March.' Then check if others behave the same.

Your mission

Spot the bias

~15 min

Quick check

A gym surveys members who still attend about satisfaction and gets 92% happy. What's the main problem?

Go deeper

1Identify the sampling method behind a survey or study you've seen recently.
2Learn survivorship bias with the classic WWII aircraft armor example.
3Practice spotting a biased sample in a real news headline about a study.