Convex Optimization on Chen Kai Blog

ML Math Derivations (4): Convex Optimization Theory

Fri, 23 Jan 2026 09:00:00 +0000

What You Will Learn#

In 1947, George Dantzig proposed the simplex method for linear programming, and a working theory of optimization was born. Eight decades later, optimization has become the engine of machine learning: every model you train, from a one-line linear regression to a 70B-parameter language model, is the answer to some optimization problem.

Essence of Linear Algebra (11): Matrix Calculus and Optimization — The Engine Behind Machine Learning

Wed, 12 Mar 2025 09:00:00 +0000

From Shower Knobs to Neural Networks#

Every morning you train a tiny neural network. The water comes out too cold, so you nudge the knob — a parameter — in some direction. A second later you observe a new temperature — the error signal — and nudge again. After three or four iterations you have converged.