Stop 05 · Machine Learning FundamentalsTopics · 8/12

Support Vector Machines

In short

SVMs find the boundary that separates classes with the widest possible gap. With kernels, they can draw curved boundaries too, which makes them strong on small and medium datasets.

A classifier that finds the maximum-margin boundary between classes, extendable to non-linear boundaries via the kernel trick — strong on smaller, high-dimensional datasets.

The mascot paints a wide road stripe with a roller between two rows of small houses.

Think of it like this

The widest road between two villages

Two villages sit on either side of a field. You must build a road between them that keeps as far as possible from both. The few houses closest to the road decide where it goes. SVM finds that widest road between two classes.

Watch it happen

Points on a line, two classes

1/5Class A sits at 1, 2 and 3. Class B sits at 7, 8 and 9.

The big ideas

What you need to know

Maximise the margin

A wide gap between classes tends to generalise better to new points.

Support vectors

Only the points nearest the boundary define it. The rest could be deleted without changing it.

Kernels bend the boundary

The RBF kernel lets SVM draw curved boundaries without you creating new features by hand.

C and gamma

C sets how much misclassification is tolerated; gamma sets how wiggly an RBF boundary can be. Tune both.

See it in code

Walk through the code

python

1from sklearn.datasets import make_moons2from sklearn.model_selection import train_test_split3from sklearn.pipeline import make_pipeline4from sklearn.preprocessing import StandardScaler5from sklearn.svm import SVC6 7X, y = make_moons(n_samples=500, noise=0.2, random_state=0)8X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=0)9 10for kernel in ["linear", "rbf"]:11    svm = make_pipeline(StandardScaler(), SVC(kernel=kernel, C=1.0))12    svm.fit(X_tr, y_tr)13    print(kernel, round(svm.score(X_te, y_te), 3))

1/3Make a curved, two-class dataset and split it.

Two interlocking half-moons can't be split by a straight line. The RBF kernel's curved boundary usually does much better.

Save yourself the pain

Common mistakes

Oops

Running SVM without scaling features.

Do this instead

SVMs rely on distances, so always scale. Put StandardScaler in the pipeline.

Oops

Using kernel SVMs on millions of rows.

Do this instead

Kernel SVM training gets very slow as data grows. Try LinearSVC or gradient boosting for big data.

Your mission

Watch the boundary bend

~30 min

Quick check

In an SVM, which training points determine the decision boundary?

Go deeper

1Train an SVM with a linear kernel and visualize the decision boundary on 2D data.
2Experiment with the RBF kernel to see how it handles non-linear separation.
3Learn the role of the C and gamma hyperparameters.