https://www.chenk.top/en/tags/acceleration/Recent content in Acceleration on Chen Kai BlogHugoenTue, 20 Sep 2022 09:00:00 +0000https://www.chenk.top/en/optimization-theory/05-acceleration-beyond-nesterov/Tue, 20 Sep 2022 09:00:00 +0000https://www.chenk.top/en/optimization-theory/05-acceleration-beyond-nesterov/<p>Article 02 introduced Nesterov acceleration and showed it improves the per-iteration cost from <span class="math-inline">$\kappa$</span> to <span class="math-inline">$\sqrt{\kappa}$</span> . This article asks the deeper questions:</p> <ul> <li><strong>Why <span class="math-inline">$\sqrt{\kappa}$</span> and not faster?</strong> We prove a matching lower bound — no first-order method can do better.</li> <li><strong>Is Nesterov the only way?</strong> Polyak&rsquo;s Heavy-Ball method achieves the same rate using a completely different update rule.</li> <li><strong>Can we accelerate any solver?</strong> The Catalyst framework wraps a black-box optimizer to gain the accelerated rate, at the cost of solving a regularized subproblem.</li> </ul> <p>The unifying tool is a <strong>Lyapunov potential</strong> — a non-negative quantity that the algorithm decreases at every step. Both Nesterov and Heavy-Ball admit Lyapunov proofs, and the lower bound essentially says no Lyapunov decrease can happen faster.</p>