ML Math Derivations (2): Linear Algebra and Matrix Theory

Wed, 21 Jan 2026 09:00:00 +0000

Why this chapter, and what’s different#

If you have already worked through a standard linear-algebra course you have seen most of these objects. This chapter is not that course. It is the ML practitioner’s slice of linear algebra: the half-dozen ideas that actually appear when you implement gradient descent, run PCA, train a neural net, or read a paper.

Essence of Linear Algebra (9): Singular Value Decomposition — The Crown Jewel of Linear Algebra

Wed, 26 Feb 2025 09:00:00 +0000

Why SVD Earns the Crown#

The spectral theorem of Chapter 8 gave us $A = Q\Lambda Q^T$ — a beautifully clean factorisation, but only for symmetric matrices. Most matrices that show up in practice are not symmetric, and many are not even square:

a photograph stored as a $1920 \times 1080$ pixel matrix,
a Netflix-style user–movie rating matrix (millions of rows, thousands of columns),
a document–term matrix in NLP (documents by vocabulary),
a gene-expression matrix in bioinformatics.

A = U\,\Sigma\,V^{\!\top}.

This is the most powerful, most universally applicable decomposition in all of linear algebra.

SVD on Chen Kai Blog

ML Math Derivations (2): Linear Algebra and Matrix Theory

Why this chapter, and what’s different#

Essence of Linear Algebra (9): Singular Value Decomposition — The Crown Jewel of Linear Algebra

Why SVD Earns the Crown#