Stop 02 · Math & Statistics for AITopics · 3/13

Eigenvalues & Eigenvectors

In short

PCA, one of the most used tricks for shrinking big datasets, is built on eigenvectors. They show the directions a matrix only stretches, which tells you where your data varies most.

Special vectors that only get scaled (not rotated) by a matrix transformation — the math underlying PCA, dimensionality reduction and understanding data variance.

The purple blob rolls out dough with a rolling pin, a straight line on it stretching longer.

Think of it like this

Rolling out a roti

Draw a few lines on a ball of dough and roll it in one direction. Most lines tilt and twist. But the line along the rolling direction just gets longer, it never turns. That line is an eigenvector, and how much longer it got is the eigenvalue.

Watch it happen

Apply A = [[2, 1], [1, 2]]

Vector [1, 0]

Becomes [2, 1]
Changes direction, tilts upward
No single stretch factor

Eigenvector [1, 1]

Becomes [3, 3]
Same direction, just longer
Stretch factor 3 is the eigenvalue

Eigenvectors are the directions a matrix keeps. In PCA, the biggest eigenvalue marks where your data spreads the most.

The big ideas

What you need to know

A v = λ v

Multiply the eigenvector v by the matrix A and you get the same v scaled by a number λ, the eigenvalue.

Eigenvalue = stretch amount

λ = 3 triples the length, λ = 0.5 halves it, a negative λ flips the direction.

PCA uses them

PCA finds eigenvectors of the data's covariance matrix. The top ones become new axes that keep most of the variation.

Let numpy do it

np.linalg.eig gives eigenvalues and eigenvectors. You need the idea more than hand calculation.

See it in code

Walk through the code

python

1import numpy as np2 3A = np.array([[2, 1],4              [1, 2]])5vals, vecs = np.linalg.eig(A)6print(vals)                 # [3. 1.]  (order can vary)7 8v = vecs[:, 0]              # eigenvector for 3, about [0.707, 0.707]9print(A @ v)                # [2.121 2.121]10print(3 * v)                # [2.121 2.121]  same thing11 12u = np.array([1, 0])13print(A @ u)                # [2 1]  direction changed

1/3eig returns all eigenvalues and a matrix whose columns are the eigenvectors.

numpy returns unit-length eigenvectors as columns. A @ v equals 3 * v, which is the whole definition in action.

Save yourself the pain

Common mistakes

Oops

Reading eigenvectors from the rows of the result.

Do this instead

numpy puts each eigenvector in a column: vecs[:, i] goes with vals[i].

Oops

Thinking eigenvectors have one correct length.

Do this instead

Any scaled copy is also an eigenvector. Libraries return length 1 by convention.

Your mission

Spot the stretch directions

~20 min

Quick check

For a matrix A, A @ v gives exactly 2 * v. What does that tell you?

Go deeper

1Compute eigenvalues/eigenvectors for a small 2x2 matrix by hand.
2Learn how PCA uses eigenvectors of the covariance matrix to find directions of maximum variance.
3Visualize how eigenvectors define the axes a linear transformation stretches along.