Stop 05 · Machine Learning FundamentalsTopics · 6/12

K-Nearest Neighbors

In short

KNN might be the simplest model there is: to predict something about a new point, look at the most similar examples you already know. Simple, but it teaches you how distance drives many ML ideas.

A simple, non-parametric algorithm that classifies a point based on the majority label of its K closest neighbors — intuitive, but slow and sensitive to feature scale at prediction time.

The mascot knocks on a neighbour's door in an apartment corridor, holding a notepad.

Think of it like this

Asking the neighbours about rent

Wondering what a flat should cost? Ask the five closest neighbours what they pay and take the average. Flats far away don't count. KNN predicts the same way: find the k most similar rows and average or vote.

Watch it happen

Rents of nearby flats (₹ thousands), nearest first

1/5Every known flat, sorted by distance to the new flat. Closest on the left.

The big ideas

What you need to know

Distance decides everything

KNN uses distances between rows, so features must be scaled or the biggest-number feature dominates.

Choosing k

Small k follows noise, large k blurs real patterns. Pick k with cross-validation.

Lazy learner

There's no real training step. All the work happens at prediction time, which gets slow on big data.

Struggles in high dimensions

With many features, everything looks far from everything. Reduce features first if needed.

See it in code

Walk through the code

python

1from sklearn.datasets import load_wine2from sklearn.model_selection import cross_val_score3from sklearn.pipeline import make_pipeline4from sklearn.preprocessing import StandardScaler5from sklearn.neighbors import KNeighborsClassifier6 7X, y = load_wine(return_X_y=True)8 9for k in [1, 5, 15]:10    knn = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=k))11    print(k, cross_val_score(knn, X, y, cv=5).mean().round(3))

1/313 chemical measurements of wines from three classes.

Scaling inside the pipeline is not optional for KNN. Compare a few k values with cross-validation.

Save yourself the pain

Common mistakes

Oops

Running KNN on unscaled features like income and age together.

Do this instead

Scale first. Otherwise income in lakhs swamps age, and 'nearest' means 'similar income' only.

Oops

Using KNN on millions of rows needing instant predictions.

Do this instead

Every prediction compares against stored rows. Use tree models, or approximate neighbour search libraries.

Your mission

Find the best k

~20 min

Quick check

Why is feature scaling so important for KNN?

Go deeper

1Implement KNN from scratch to understand distance-based classification.
2Experiment with different values of K and observe the bias-variance tradeoff.
3Learn why feature scaling matters critically for distance-based algorithms like KNN.