Stop 06 · Machine Learning FundamentalsTopics · 12/13

Support Vector Machines

In short

SVMs find the boundary that separates classes with the widest possible gap. With kernels, they can draw curved boundaries too, which makes them strong on small and medium datasets.

A model that finds the best boundary (hyperplane) separating classes with the widest possible margin — powerful for certain problems, especially with the 'kernel trick' for non-linear boundaries.

The mascot paints a wide road stripe with a roller between two rows of small houses.

Think of it like this

The widest road between two villages

Two villages sit on either side of a field. You must build a road between them that keeps as far as possible from both. The few houses closest to the road decide where it goes. SVM finds that widest road between two classes.

Watch it happen

Points on a line, two classes

1/5Class A sits at 1, 2 and 3. Class B sits at 7, 8 and 9.

The big ideas

What you need to know

Maximise the margin

A wide gap between classes tends to generalise better to new points.

Support vectors

Only the points nearest the boundary define it. The rest could be deleted without changing it.

Kernels bend the boundary

The RBF kernel lets SVM draw curved boundaries without you creating new features by hand.

C and gamma

C sets how much misclassification is tolerated; gamma sets how wiggly an RBF boundary can be. Tune both.

See it in code

Walk through the code

python

1from sklearn.datasets import make_moons2from sklearn.model_selection import train_test_split3from sklearn.pipeline import make_pipeline4from sklearn.preprocessing import StandardScaler5from sklearn.svm import SVC6 7X, y = make_moons(n_samples=500, noise=0.2, random_state=0)8X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=0)9 10for kernel in ["linear", "rbf"]:11    svm = make_pipeline(StandardScaler(), SVC(kernel=kernel, C=1.0))12    svm.fit(X_tr, y_tr)13    print(kernel, round(svm.score(X_te, y_te), 3))

1/3Make a curved, two-class dataset and split it.

Two interlocking half-moons can't be split by a straight line. The RBF kernel's curved boundary usually does much better.

Save yourself the pain

Common mistakes

Oops

Running SVM without scaling features.

Do this instead

SVMs rely on distances, so always scale. Put StandardScaler in the pipeline.

Oops

Using kernel SVMs on millions of rows.

Do this instead

Kernel SVM training gets very slow as data grows. Try LinearSVC or gradient boosting for big data.

Your mission

Watch the boundary bend

~30 min

Quick check

In an SVM, which training points determine the decision boundary?

Go deeper

1Fit an SVM on a simple 2D dataset and visualize the decision boundary.
2Learn the intuition behind the 'kernel trick' without needing the full math.
3Understand why SVMs can be slow to train on very large datasets.