Stop 18 · Evaluation, Safety & Responsible AITopics · 3/9

Bias & Fairness in AI

In short

Models learn from past data, and past data carries past unfairness. A loan or hiring model can quietly treat groups differently. You have to measure it, because it won't announce itself.

Systematic, unfair skew in model outputs across groups — often inherited from training data — a real risk that requires deliberate testing, not assumption of neutrality.

The blob with binoculars watches a cricket game while a row of other players waits unnoticed behind.

Think of it like this

The selector who watched one city

Imagine a cricket selector who only ever watched matches in one city. He'd pick great players, but almost all from that city, and miss talent everywhere else. He isn't evil, his data was narrow. Models trained on narrow or skewed data behave the same way.

Watch it happen

Selection rate by group (example data)

Group A: 60%

Group B: 42%

Group C: 58%

1/5A shortlisting model is run on 3,000 applicants. We split the results by a sensitive attribute.

The big ideas

What you need to know

Bias starts in data

Skewed samples, historical decisions and missing groups all teach the model unfair patterns.

Removing the column isn't enough

Drop gender and the model may still learn it from proxies like hobbies, names or pincode.

Pick a fairness metric

Equal selection rates, equal error rates and equal precision can't all hold at once. Choose what fits the decision.

Test by slice

Always report accuracy and errors for each group, not just one overall number.

See it in code

Walk through the code

python

1import pandas as pd2 3df = pd.DataFrame({4    "group":       ["A", "A", "A", "B", "B", "B", "B", "A"],5    "shortlisted": [1,   1,   0,   0,   1,   0,   0,   1],6})7 8rates = df.groupby("group")["shortlisted"].mean()9print(rates)10 11ratio = rates.min() / rates.max()12print(f"Selection rate ratio: {ratio:.2f}")13if ratio < 0.8:14    print("Flag for review: one group is selected much less often")

1/3Each row is one applicant: their group and whether the model shortlisted them.

A first fairness check: selection rate per group and the ratio between them. Libraries like Fairlearn add many more metrics.

Save yourself the pain

Common mistakes

Oops

Reporting one overall accuracy of 92% and calling the model fair.

Do this instead

Break metrics down by group. 92% overall can hide 70% for a smaller group.

Oops

Assuming an LLM is neutral because nobody programmed bias into it.

Do this instead

LLMs learn from internet text, which has stereotypes. Test outputs across names, genders and regions.

Your mission

Probe an LLM for bias

~30 min

Quick check

You removed the gender column, but your model still shortlists women less. What's a likely cause?

Go deeper

1Test a model with prompts that vary only demographic-related terms and compare outputs.
2Learn common sources of bias: training data imbalance, labeling bias, proxy variables.
3Read a case study of a real AI fairness failure and what caused it.