Eigenvectors: Directions That Refuse to Turn

Most vectors change both length and direction under a matrix. An eigenvector is special:

Av=\lambda v.

It may stretch, shrink or reverse, but it remains on the same line.

A transformed grid with two highlighted invariant eigenvector directions. — Amid all the distortion, the eigen-directions refuse to turn.

Why should such directions exist?

Imagine repeatedly applying a transformation. Components along different eigenvectors are repeatedly multiplied by their eigenvalues.

Repeated transformation causing a vector to align with a dominant eigenvector. — Repeated application often makes the component associated with the largest-magnitude eigenvalue dominate.

This is the intuition behind the power method.

Why ML cares

For a covariance matrix, eigenvectors identify principal directions of variation. PCA orders these directions by eigenvalue, letting us preserve the most important variation with fewer coordinates.

In dynamics, eigenvalues tell us whether modes grow or decay. In quantum mechanics, eigenvectors and eigenvalues become states and observable values.

The same tiny equation $Av=\lambda v$ appears everywhere because it discovers the transformation’s own preferred coordinate system.