Stop 22 · Interview PreparationTechnical · 8/8

Debugging Model Performance Issues

In short

"Your model's accuracy dropped. What do you do?" Interviewers want a calm, ordered process, not a list of random tricks. Data first, then the model, with evidence at every step.

A practical skill interviewers test directly — given a model performing poorly, can you systematically narrow down whether it's data, features, architecture, or training issues.

The purple blob kneels beside a scooter, checking it with a torch next to an open toolbox.

Think of it like this

The scooter mechanic

A good mechanic doesn't open the engine first when your scooter won't start. Fuel? Spark plug? Battery? Simple, likely causes get checked before the expensive ones. Debugging a model is the same: check the data and the pipeline before blaming the architecture.

Watch it happen

Where is it failing?

Model is bad

Train error high

Val error high

Prod worse

Underfit or bug

Overfit or leak

Drift or skew

1/4First compare against a simple baseline. Then ask where the model fails: training, validation, or only in production.

The big ideas

What you need to know

Baseline first

A majority-class guess or simple linear model tells you if your fancy model is adding anything.

Check the data before the model

Wrong labels, duplicates across splits, missing values and leaked target columns cause most surprises.

Error analysis by slice

Group mistakes by city, device or class. A model is often fine overall and terrible on one slice.

Learning curves tell the story

Plot train and validation loss. A wide gap means overfitting; both high means underfitting.

See it in code

Walk through the code

python

1import pandas as pd2 3df = pd.read_csv("val_predictions.csv")   # columns: city, device, label, pred4df["correct"] = df["label"] == df["pred"]5 6print("overall:", round(df["correct"].mean(), 3))7by_slice = (df.groupby(["city", "device"])["correct"]8              .agg(accuracy="mean", n="size")9              .sort_values("accuracy"))10print(by_slice.head(5))   # the worst slices are where to look first

1/3Load saved validation predictions and mark each row right or wrong.

Slice-based error analysis in a few lines. It often reveals that "the model is bad" really means "the model is bad on old Android phones in one city".

Save yourself the pain

Common mistakes

Oops

Answering "I'd try a bigger model" or "I'd tune hyperparameters" first.

Do this instead

Start with data and pipeline checks, compare with a baseline, then look at where errors happen.

Oops

Trusting a suspiciously great validation score.

Do this instead

Look for leakage: a feature that's only known after the outcome, or the same user in train and test.

Your mission

Break a model, then debug it

~45 min

Quick check

A model scores 99% on validation but 70% in production. Which cause should you suspect first?

Go deeper

1Build a mental checklist: data quality, label quality, feature leakage, overfitting, underfitting.
2Practice narrating your debugging process out loud, not just the final fix.
3Review a real bug you hit in a project and be ready to tell that story in an interview.

More technical at this stop