Build & Compare 3 ML Models on the Same Dataset
Part of the Data Science Roadmap
In short
Knowing algorithms is good. Showing you can pick one fairly, with the same data and the same tests for all, is what interviewers really look for. This project proves exactly that.
The core practice project for this phase — train a linear/logistic baseline, a tree-based model, and an ensemble model on the same problem, and compare their performance honestly.
Think of it like this
Test-riding three scooters
Before buying, you ride three scooters on the same road, same day, same passenger. Then you compare mileage, comfort and price, not just top speed. Comparing models works the same way: same data, same folds, several metrics.
Watch it happen
Your build plan
1/6Pick a classification dataset, clean it and decide the metric that matters, like recall.
The big ideas
What you need to know
Same folds, same pipeline
A fair comparison uses identical splits and preprocessing for every model.
Pick metrics first
Decide on accuracy, recall or MAE before seeing results, so you can't cherry-pick.
Always include a baseline
sklearn's DummyClassifier shows what 'no skill' scores. It keeps your results honest.
Accuracy isn't everything
Training time, prediction speed and explainability matter too. Note them in your table.
See it in code
Walk through the code
1from sklearn.datasets import load_breast_cancer2from sklearn.model_selection import cross_validate, StratifiedKFold3from sklearn.pipeline import make_pipeline4from sklearn.preprocessing import StandardScaler5from sklearn.linear_model import LogisticRegression6from sklearn.ensemble import RandomForestClassifier7from sklearn.neighbors import KNeighborsClassifier8X, y = load_breast_cancer(return_X_y=True)9cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)10models = {"logreg": make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)),11 "forest": RandomForestClassifier(random_state=0),12 "knn": make_pipeline(StandardScaler(), KNeighborsClassifier())}13for name, m in models.items():14 r = cross_validate(m, X, y, cv=cv, scoring=["accuracy", "recall"])15 print(name, r["test_accuracy"].mean().round(3), r["test_recall"].mean().round(3))1/4Load data and fix one set of shuffled, stratified folds for everyone.
The core of the project: three models, one set of folds, two metrics. Add a DummyClassifier as a fourth row for the baseline.
Save yourself the pain
Common mistakes
Oops
Tuning one model heavily and leaving the others on defaults.
Do this instead
Give each model a similar, modest tuning budget, or the comparison is unfair.
Oops
Declaring a winner on a 0.2% difference.
Do this instead
Check the spread across folds. Tiny gaps are often noise, so weigh simplicity and speed too.
Your mission
First milestone
Quick check
Why use the same cross-validation folds for every model you compare?
Go deeper
- 1Pick a clean, well-known dataset so you can focus on modeling, not cleaning.
- 2Use the same train/test split and evaluation metric across all 3 models for a fair comparison.
- 3Write a short summary of which model won, and your hypothesis for why.
Stuck on this topic? Ask an Insider
Get 1:1 guidance from people who've walked this exact path — on the InsideEdge app.
