Convex Analysis
Sep 26, 2022
Optimization Theory
22 min readOptimization (9): Interior-Point Methods and Self-Concordant Barriers
How interior-point methods became the default solver for convex programming: replace inequalities with a logarithmic barrier, parametrize the central path, and apply Newton's method. Includes the self-concordance …
Sep 24, 2022
Optimization Theory
20 min readOptimization (8): Lagrangian Duality and KKT Conditions
How constraints become prices: the Lagrangian, weak duality, Slater's condition for strong duality, the KKT system as necessary and sufficient optimality, and why the SVM dual is much smaller than the SVM primal. …
Sep 21, 2022
Optimization Theory
32 min readOptimization (6): Composite Optimization and Proximal Methods
A systematic walk through the proximal operator: convex-analysis basics, the Moreau envelope, closed-form proxes, and how they power ISTA, FISTA, ADMM, LASSO, and SVM in practice.
Sep 15, 2022
Optimization Theory
28 min readOptimization (2): Smoothness, Strong Convexity, and Nesterov Acceleration
Three concepts that demystify most of optimization: Lipschitz smoothness fixes the maximum step size, strong convexity sets the convergence rate and uniqueness of the minimizer, and Nesterov acceleration replaces kappa …
Sep 14, 2022
Optimization Theory
26 min readOptimization (1): Convex Analysis Foundations
The geometric and analytic toolkit that unlocks the rest of the series: convex sets, convex functions, the conjugate (Fenchel) transform, subgradients, and the indicator/support function pair. Includes complete proofs of …