Model Evaluation Questions
Part of the Data Science Roadmap
In short
'Why not just use accuracy?' is one of the most common interview questions in data science. The real test: can you pick a metric based on what each kind of mistake costs?
'Why wouldn't you use accuracy for this problem?' or 'walk me through choosing a metric for a fraud detection model' — testing Phase 7's evaluation judgment against a specific business scenario.
Think of it like this
The airport metal detector
Set the detector too sensitive and every belt buckle beeps: false alarms, long queues. Set it too loose and something dangerous walks through. Choosing a metric and threshold means choosing which mistake you can live with. Interviewers want to hear you reason about that cost.
Watch it happen
Moving the fraud threshold
1/5Eight transactions sorted by fraud score. F marks the four real frauds.
The big ideas
What you need to know
Accuracy hides rare classes
If 1% of transactions are fraud, predicting 'never fraud' is 99% accurate and completely useless.
Precision vs recall
Precision: of what you flagged, how much was right. Recall: of what was really there, how much you caught.
Threshold is a business choice
0.5 is just a default. Pick the cut-off from the cost of each error type.
Offline vs online
Good offline metrics are a start. Confirm real impact with an A/B test after launch.
See it in code
Walk through the code
1import numpy as np2from sklearn.metrics import precision_score, recall_score3 4y_true = np.array([1, 1, 0, 1, 0, 0, 1, 0])5scores = np.array([0.95, 0.90, 0.80, 0.70, 0.60, 0.40, 0.30, 0.20])6 7for t in [0.75, 0.5, 0.25]:8 y_pred = (scores >= t).astype(int)9 p = precision_score(y_true, y_pred)10 r = recall_score(y_true, y_pred)11 print(f"threshold {t}: precision {p:.2f}, recall {r:.2f}")1/3True labels (1 = fraud) and the model's scores, sorted high to low.
The same eight transactions as the animation. Lower thresholds catch more fraud but flag more innocent people.
Save yourself the pain
Common mistakes
Oops
Answering 'I'd use accuracy' for fraud, churn or disease detection.
Do this instead
Ask about class balance and error costs, then suggest precision, recall, F1 or PR-AUC with reasons.
Oops
Keeping the default 0.5 threshold without thinking.
Do this instead
Pick the threshold from business costs or capacity, like 'the team can review 200 alerts a day'.
Your mission
Pick the metric
Quick check
In a cancer screening model, missing a real case is far worse than a false alarm. What should you prioritise?
Go deeper
- 1Review Phase 7 and practice matching evaluation metrics to different business scenarios.
- 2Prepare to explain precision-recall trade-offs concretely, with a real example.
- 3Practice justifying a metric choice, not just naming metrics you know.
More ml & statistics at this stop
ML Algorithm Deep-Dive Questions
Statistics & Probability Questions
Stuck on this topic? Ask an Insider
Get 1:1 guidance from people who've walked this exact path — on the InsideEdge app.
