Stop 06 · Feature Engineering & Model EvaluationTopics · 8/12

Evaluation Metrics (Accuracy, Precision, Recall, F1)

In short

'95% accurate' sounds amazing until you learn only 5% of the emails were spam. Picking the right metric tells you what your model is actually good and bad at.

The core classification metrics — accuracy alone is misleading on imbalanced data, so precision, recall and F1 tell you what kind of errors your model actually makes.

The blob as a security guard holds a beeping metal detector wand at a mall entrance gate.

Think of it like this

The mall metal detector

A mall entrance detector beeps at people. Precision asks: of all the beeps, how many really had metal? Recall asks: of everyone carrying metal, how many made it beep? A detector that beeps at everyone has perfect recall and awful precision.

Watch it happen

Scoring a spam filter on 10 emails

flag

1/5The model's verdict on 10 emails. In reality, emails 0, 1, 3 and 8 are spam.

The big ideas

What you need to know

Accuracy: share of all correct

Correct predictions divided by all predictions. Misleading when one class is much rarer than the other.

Precision: trust in a flag

Of everything the model called positive, how much really was? Matters when false alarms are expensive.

Recall: how many you caught

Of all real positives, how many did the model find? Matters when missing one is costly, like cancer screening.

F1 balances both

F1 is the harmonic mean of precision and recall. It drops sharply if either one is low.

See it in code

Walk through the code

python

1from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score2 3y_true = [1, 1, 0, 1, 0, 0, 0, 0, 1, 0]  # 1 = really spam4y_pred = [1, 1, 0, 1, 0, 0, 1, 0, 0, 0]  # 1 = model flagged it5 6print("accuracy: ", accuracy_score(y_true, y_pred))   # 0.87print("precision:", precision_score(y_true, y_pred))  # 0.758print("recall:   ", recall_score(y_true, y_pred))     # 0.759print("f1:       ", f1_score(y_true, y_pred))         # 0.75

1/4y_true holds the real answers, y_pred holds the model's guesses, in the same order.

The same ten emails from the scene. Change one prediction at a time and watch which metric moves.

Save yourself the pain

Common mistakes

Oops

Reporting only accuracy on a fraud dataset where 1% of rows are fraud.

Do this instead

A model that says 'never fraud' gets 99%. Report precision and recall for the fraud class too.

Oops

Chasing the highest F1 without asking which error hurts more.

Do this instead

Ask the business: is a false alarm or a miss worse? Then lean towards precision or recall.

Your mission

Break the metrics on purpose

~15 min

Quick check

A cancer screening test should miss as few real cases as possible. Which metric should you push up first?

Go deeper

1Compute all four metrics by hand from a confusion matrix on a small example.
2Learn when to optimize for precision (spam) vs. recall (disease detection).
3Practice reading scikit-learn's `classification_report` output fluently.