ML Math Derivations (17): Dimensionality Reduction and PCA

Thu, 05 Feb 2026 09:00:00 +0000

What You Will Learn#

Feed a clustering algorithm $$10{,}000$$ -dimensional data and it will most likely fail — not because the algorithm is broken, but because high-dimensional space is a hostile environment for distance-based learning. Volumes evaporate into thin shells, the ratio of nearest- to farthest-neighbour distances tends to $$1$$ , and “closeness” stops carrying information. Dimensionality reduction is the response: project the data into a lower-dimensional space while keeping the structure that actually matters.

Essence of Linear Algebra (9): Singular Value Decomposition — The Crown Jewel of Linear Algebra

Wed, 26 Feb 2025 09:00:00 +0000

Why SVD Earns the Crown#

The spectral theorem of Chapter 8 gave us $A = Q\Lambda Q^T$ — a beautifully clean factorisation, but only for symmetric matrices. Most matrices that show up in practice are not symmetric, and many are not even square:

a photograph stored as a $1920 \times 1080$ pixel matrix,
a Netflix-style user–movie rating matrix (millions of rows, thousands of columns),
a document–term matrix in NLP (documents by vocabulary),
a gene-expression matrix in bioinformatics.

A = U\,\Sigma\,V^{\!\top}.

This is the most powerful, most universally applicable decomposition in all of linear algebra.

Dimensionality Reduction on Chen Kai Blog

ML Math Derivations (17): Dimensionality Reduction and PCA

What You Will Learn#

Essence of Linear Algebra (9): Singular Value Decomposition — The Crown Jewel of Linear Algebra

Why SVD Earns the Crown#