Stop 06 · Feature Engineering & Model EvaluationTopics · 9/12

ROC-AUC

In short

Most classifiers give a probability, and you pick the cutoff. ROC-AUC tells you how good the model is across every possible cutoff, before you commit to one.

A curve and single-number summary (Area Under Curve) of a binary classifier's performance across all classification thresholds — useful for comparing models independent of a chosen threshold.

The blob slides a big lever up and down beside a line of students standing in order of height.

Think of it like this

Setting the admission cutoff

A college can set its cutoff at 95% or 80%. A lower cutoff admits more strong students, but also more weak ones. The ROC curve shows that trade-off at every cutoff. AUC asks: pick one strong and one weak applicant, how often does the score rank the strong one higher?

Watch it happen

Moving the cutoff over 10 scores

0.95 +

0.90 +

0.80 -

0.70 +

0.60 -

0.45 +

0.40 -

0.30 -

0.20 -

0.10 -

1/5Ten predictions sorted by score. + means really positive, - means really negative. Four are positive.

The big ideas

What you need to know

ROC plots TPR vs FPR

Each point is one cutoff. Up means catching more positives. Right means more false alarms.

AUC is a ranking score

AUC is the chance a random positive gets a higher score than a random negative. 0.5 is coin-flip, 1.0 is perfect.

Threshold-free comparison

AUC lets you compare two models without picking a cutoff first. Choose the cutoff later based on business costs.

Rare positives? Check PR too

With very few positives, AUC can look rosy. A precision-recall curve shows how many flags are actually right.

See it in code

Walk through the code

python

1from sklearn.metrics import roc_auc_score, roc_curve2 3y_true  = [1, 1, 0, 1, 0, 1, 0, 0, 0, 0]4y_score = [0.95, 0.90, 0.80, 0.70, 0.60, 0.45, 0.40, 0.30, 0.20, 0.10]5 6print("AUC:", roc_auc_score(y_true, y_score))   # 0.8757 8fpr, tpr, thresholds = roc_curve(y_true, y_score)9for f, t, th in zip(fpr, tpr, thresholds):10    print(f"cutoff {th:.2f}: TPR {t:.2f}, FPR {f:.2f}")

1/3The same ten predictions from the scene: real labels and the model's probability scores.

AUC needs scores, not 0/1 predictions. In a real model, use predict_proba(X)[:, 1] as y_score.

Save yourself the pain

Common mistakes

Oops

Passing model.predict(X) to roc_auc_score.

Do this instead

predict gives hard 0/1 labels. Pass predict_proba(X)[:, 1] so AUC can see the full ranking.

Oops

Thinking an AUC of 0.9 means 90% accuracy.

Do this instead

AUC measures ranking quality across all cutoffs. Accuracy depends on the one cutoff you pick.

Your mission

Draw your own ROC curve

~25 min

Quick check

A model has an ROC-AUC of 0.5. What does that tell you?

Go deeper

1Plot an ROC curve for a trained classifier and interpret the AUC value.
2Learn the difference between ROC-AUC and precision-recall AUC, and when each is more informative.
3Understand what AUC = 0.5 vs. AUC = 1.0 actually mean.