Stop 07 · Model Evaluation & TuningTopics · 3/8

ROC-AUC Curve

In short

Most classifiers give a probability, and you pick the cut-off. The ROC curve shows every cut-off at once, and AUC squeezes it into one number you can compare.

A curve plotting true positive rate against false positive rate across different classification thresholds — AUC (area under the curve) summarizes overall model discriminative power in one number.

The mascot turns a sensitivity knob on a walk-through metal detector, with keys in a tray beside it.

Think of it like this

The metal detector knob

A mall's metal detector has a sensitivity knob. Turn it up and it catches every knife, but also beeps at belt buckles. Turn it down and fewer false beeps, but something might slip through. ROC shows what you get at every position of that knob.

Watch it happen

Moving the threshold over 8 scores

P .95

P .90

N .80

P .70

N .60

P .40

N .30

N .20

1/5Model scores for 8 cases, sorted high to low. P = really positive, N = really negative. 4 of each.

The big ideas

What you need to know

TPR vs FPR

The y-axis is recall (true positive rate). The x-axis is the share of negatives wrongly flagged.

AUC from 0.5 to 1

0.5 is coin-toss guessing, 1.0 is perfect ranking. Below 0.5 usually means labels are flipped somewhere.

AUC measures ranking

AUC is the chance a random positive gets a higher score than a random negative. It ignores the threshold.

Rare classes need care

With very few positives, ROC can look rosy. Check the precision-recall curve as well.

See it in code

Walk through the code

python

1from sklearn.datasets import load_breast_cancer2from sklearn.linear_model import LogisticRegression3from sklearn.model_selection import train_test_split4from sklearn.metrics import roc_auc_score, roc_curve5 6X, y = load_breast_cancer(return_X_y=True)7X_tr, X_te, y_tr, y_te = train_test_split(X, y, stratify=y, random_state=0)8 9model = LogisticRegression(max_iter=5000).fit(X_tr, y_tr)10scores = model.predict_proba(X_te)[:, 1]11 12print("AUC:", roc_auc_score(y_te, scores))13fpr, tpr, thresholds = roc_curve(y_te, scores)

1/3Fit the model, then take column 1 of predict_proba: the probability of class 1.

ROC needs scores, not hard 0/1 predictions. predict_proba gives the probability of the positive class.

Save yourself the pain

Common mistakes

Oops

Passing model.predict() output to roc_auc_score.

Do this instead

Pass probabilities or decision scores. Hard labels give only one point, so the AUC is misleading.

Oops

Assuming a high AUC means the default 0.5 threshold is fine.

Do this instead

AUC ignores the threshold. Pick the cut-off separately, based on the errors your problem can afford.

Your mission

Plot two ROC curves

~25 min

Quick check

Model A has AUC 0.92 and model B has AUC 0.50. What does B's score suggest?

Go deeper

1Plot an ROC curve for a trained classification model.
2Learn what an AUC of 0.5 versus 1.0 versus a specific value like 0.85 actually means.
3Understand ROC-AUC can be misleading on severely imbalanced datasets — check precision-recall curves too.