ML Math Derivations (5): Linear Regression

Sat, 24 Jan 2026 09:00:00 +0000

Hook. In 1886 Francis Galton noticed something strange about heredity: children of unusually tall (or short) parents tended to be closer to the average than their parents were. He called this drift toward the mean regression, and the name stuck. The statistical curiosity grew up into the most consequential model in machine learning — not because linear regression is powerful on its own, but because almost every other algorithm (logistic regression, neural networks, kernel methods) is some twist on the same idea: fit a line, but in the right space.

Essence of Linear Algebra (7): Orthogonality and Projections — When Vectors Mind Their Own Business

Wed, 12 Feb 2025 09:00:00 +0000

Why Orthogonality Matters#

Two vectors are orthogonal when they “do not interfere” with one another. That single idea — one direction tells you nothing about the other — powers GPS positioning, noise-canceling headphones, JPEG compression, recommendation systems, and most of numerical linear algebra.

Orthogonality is the single biggest computational shortcut in linear algebra. With a generic basis, finding coordinates is solving a linear system. With an orthogonal basis, finding coordinates is one dot product per axis. Hard problem, easy problem, same problem — just a better basis.

Least Squares on Chen Kai Blog

ML Math Derivations (5): Linear Regression

Essence of Linear Algebra (7): Orthogonality and Projections — When Vectors Mind Their Own Business

Why Orthogonality Matters#