https://www.chenk.top/en/tags/vanishing-gradients/Recent content in Vanishing Gradients on Chen Kai BlogHugoenSat, 07 Feb 2026 09:00:00 +0000https://www.chenk.top/en/ml-math-derivations/19-neural-networks-and-backpropagation/Sat, 07 Feb 2026 09:00:00 +0000https://www.chenk.top/en/ml-math-derivations/19-neural-networks-and-backpropagation/<p><figure class="article-figure"> <img src="https://blog-pic-ck.oss-cn-beijing.aliyuncs.com/posts/en/ml-math-derivations/19-Neural-Networks-and-Backpropagation/illustration_1.png" alt="ML Math Derivations (19): Neural Networks and Backpropagation — Chapter overview" loading="lazy" decoding="async" class="content-image"> </figure> </p> <hr> <blockquote> <p><strong>Hook.</strong> In 1969 Minsky and Papert proved that a single perceptron could not learn XOR, and connectionist research went into a fifteen-year freeze. The thaw came when Rumelhart, Hinton and Williams realised that <em>stacking</em> perceptrons makes the problem disappear — and that the same chain rule everyone learns in calculus, applied carefully, computes every gradient in a multilayer network for the cost of a single extra forward pass. That algorithm is backpropagation. Every gradient in every Transformer, every diffusion model, every GPT trained today still runs on it.</p>