Algorithm on Chen Kai Blog

Reparameterization Trick & Gumbel-Softmax: A Deep Dive

Wed, 30 Jul 2025 09:00:00 +0000

The moment your model contains a sampling step, training hits a hard wall: how do gradients flow through a random node?

The reparameterization trick has a clean answer — rewrite $z\sim p_\theta(z)$ as $z=g_\theta(\epsilon)$ , isolating the randomness in a parameter-free noise variable $\epsilon$ , so backprop can flow through $g_\theta$ . The trouble starts with discrete variables: operations like $\arg\max$ are not differentiable. Gumbel-Softmax (a.k.a. the Concrete distribution) replaces the discrete sample with a tempered softmax over Gumbel-perturbed logits, giving you a smooth, differentiable surrogate that you can train end-to-end.

Low-Rank Matrix Approximation and the Pseudoinverse: From SVD to Regularization

Mon, 28 Jul 2025 09:00:00 +0000

Real data matrices are almost never both square and full rank: correlated features, too few samples, and noise-induced ill-conditioning all make “matrix inverse” either undefined or numerically useless. The pseudoinverse (Moore-Penrose inverse) preserves the spirit of an inverse while dropping the impossible-to-meet requirements: it redefines the “solution” of a linear system as the least-squares solution, breaking ties by picking the one with minimum norm. This post derives the pseudoinverse from that least-squares viewpoint, gives the four Penrose conditions, builds it from the SVD, and connects this single object to the Eckart-Young low-rank approximation theorem, PCA, recommender-system matrix factorization, and LoRA fine-tuning.

Time Series Forecasting (8): Informer — Efficient Long-Sequence Forecasting

Sun, 15 Dec 2024 09:00:00 +0000

The Transformer is wonderful at sequence modeling — right up to the moment your sequence gets long. Vanilla self-attention costs $\mathcal{O}(L^2)$ in both compute and memory, so a one-week hourly window (168 steps) is fine, a one-month window (720 steps) is painful, and a three-month window (2160 steps) is essentially impossible on a single GPU. That is exactly the regime real-world long-horizon forecasting lives in: weather, energy, finance, IoT.

Time Series Forecasting (7): N-BEATS — Interpretable Deep Architecture

Sat, 30 Nov 2024 09:00:00 +0000

The 2018 M4 forecasting competition served 100,000 series across six frequencies as a single benchmark. The leaderboard was dominated by hand-tuned ensembles built from decades of statistical-forecasting craft. Then a pure neural network with no statistical preprocessing, no feature engineering, and no recurrence won outright. That network was N-BEATS by Oreshkin et al. — a stack of fully-connected blocks with two residual paths. Its interpretable variant additionally split the forecast into a polynomial trend and a Fourier seasonality, so the very thing classical statisticians wanted (a readable decomposition) came for free.

Time Series Forecasting (6): Temporal Convolutional Networks (TCN)

Fri, 15 Nov 2024 09:00:00 +0000

For most of the 2010s, saying “deep learning for time series” meant using LSTM. The story changed in 2018 when Bai, Kolter, and Koltun published An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling. Their result was surprisingly simple: use a stack of 1-D convolutions, make them causal (no peeking at the future), space the filter taps exponentially (dilation), wrap the whole thing in residual connections, and train. Task after task, the resulting Temporal Convolutional Network (TCN) matched or beat LSTM/GRU — while training several times faster because every time step in the forward pass runs in parallel.

Time Series Forecasting (5): Transformer Architecture for Time Series

Thu, 31 Oct 2024 09:00:00 +0000

The 2017 Attention Is All You Need paper took the attention mechanism from the previous chapter to its logical extreme: drop the RNN entirely. Transformers stack pure attention into a full sequence model — no recurrence, no hidden state propagating over time. Originally designed for machine translation, the architecture was quickly adapted to every other sequence task, time series included.

Dropping a vanilla NLP Transformer onto a time-series problem runs into two immediate complications. The first is position. Attention is a set operation — shuffle the input order and the output is unchanged. For a time series, order is everything: a temperature curve that goes up-then-down and one that goes down-then-up are entirely different signals. NLP solves this with sinusoidal position encodings; do those still make sense for time series, or should we use learned encodings, or just concatenate calendar features (hour-of-day, day-of-week) directly into the input?

Time Series Forecasting (4): Attention Mechanisms — Direct Long-Range Dependencies

Wed, 16 Oct 2024 09:00:00 +0000

RNNs and LSTMs handled “too many time steps” but left a subtler limitation in place: information has to travel step by step. For step 100 to see what happened at step 1, the signal has to ride the hidden state through 99 intermediate stops — and each stop attenuates the signal a little and squashes it through a nonlinearity. Even with LSTM’s “highway” cell state, it’s still a single lane in a single direction.

Time Series Forecasting (3): GRU — Lightweight Gates and Efficiency Trade-offs

Tue, 01 Oct 2024 09:00:00 +0000

After you’ve used LSTM for a while, an obvious question shows up: aren’t three gates a bit much? The forget and input gates seem to do related work — one decides what to drop, the other decides what to add — couldn’t they be merged? And does the cell state really need to be a separate vector from the hidden state, or could the hidden state do double duty?

That is exactly the question Cho et al. answered in 2014 with the Gated Recurrent Unit. They collapsed three gates into two: an update gate that controls how much of the old state to keep versus how much new content to absorb, and a reset gate that decides whether to ignore the old state entirely when computing a fresh candidate. The cell state is folded back into the hidden state. The result is roughly 25% fewer parameters, training that runs 10-15% faster, and accuracy on most time-series tasks that is statistically indistinguishable from LSTM.

Time Series Forecasting (2): LSTM — Gate Mechanisms and Long-Term Dependencies

Mon, 16 Sep 2024 09:00:00 +0000

The first RNN I ever trained, back in 2017, was a small sales forecaster: 50 days in, the next day out. The forward pass ran cleanly, the loss went down, and yet the model had near-total amnesia about anything older than three days. The data had a clear monthly cycle. The model couldn’t see it. I assumed I needed more data, so I added rows and layers — and watched the training loss jump to NaN halfway through epoch two.

Time Series Forecasting (1): Traditional Statistical Models

Sun, 01 Sep 2024 09:00:00 +0000

The first time I touched data that “looked like a time series” — hourly server CPU usage — my instinct was to throw it at a linear regression. Time on the x-axis, usage on the y-axis. The fit was terrible. The problem wasn’t the regression; the problem was that this kind of data has its own personality. It has trends, seasonality, and a stubborn dependence between consecutive observations. A vanilla regression treats every row as an independent sample and throws away the one piece of information that matters most: time itself.

Position Encoding Brief: From Sinusoidal to RoPE and ALiBi

Fri, 30 Jun 2023 09:00:00 +0000

Self-attention has a strange property that surprises most people the first time they compute it by hand: it does not know the order of its inputs. Permute the tokens and every attention score is permuted along with them — the function is exactly equivariant. So before we can do anything useful with a Transformer, we have to inject position information from the outside.

That single design decision — how to inject it — has spawned a remarkable amount of research. Sinusoidal, learned, relative, T5-style buckets, RoPE, ALiBi, NoPE, and more. This post is a practitioner’s brief: enough math to know why each scheme works, enough comparison to choose one, and a clear focus on the property that matters most in the LLM era — length extrapolation, the ability to handle sequences longer than anything seen in training.

Variational Autoencoder (VAE): From Intuition to Implementation and Troubleshooting

Tue, 27 Jun 2023 09:00:00 +0000

A plain autoencoder compresses and reconstructs. A variational autoencoder learns something far more useful: a smooth, structured latent space you can sample from to generate genuinely new data. That single change — making the encoder output a distribution instead of a vector — turns the network from a fancy compressor into a generative model with a tractable likelihood lower bound.

This guide walks the full path: why autoencoders fail at generation, how the ELBO derivation gets you to the loss function, why the reparameterization trick is the trick that makes everything trainable, a complete PyTorch implementation, and a tour of every common failure mode with concrete fixes.

Optimization (12): Discrete and Global Optimization

Fri, 30 Sep 2022 09:00:00 +0000

The first eleven articles in this series tackled continuous convex problems (or convex relaxations of non-convex ones). This final article addresses two harder regimes:

Discrete optimization: variables take integer or combinatorial values. The feasible set is a finite (but exponentially large) collection of points. Linear and convex tools no longer apply directly because there are no derivatives across the integer lattice.
Global non-convex optimization: variables are continuous but the function has many local minima, and we want the global one. Methods like Newton and L-BFGS only find local minima.

Both regimes share a key feature: provably optimal algorithms are exponential in the worst case. Practical solutions come from (a) exact algorithms with smart pruning (branch-and-bound) and (b) heuristics that find good (but not optimal) solutions in polynomial time.

Optimization (11): Non-Convex Optimization and Saddle Escape

Thu, 29 Sep 2022 09:00:00 +0000

For non-convex $$f$$ , gradient descent has no global guarantee. The best we can say is that $\nabla f(x_t) \to 0$ — we converge to a stationary point, which could be a local min, a saddle, or even a local max. This article asks: when can we say more?

Three positive results:

Saddle escape: under a “strict saddle” assumption, perturbed GD converges to local minima in polynomial time. Saddle points are unstable; Brownian noise (or just numerical perturbation) escapes them.
PL condition: a relaxation of strong convexity that holds in over-parameterized neural networks. Under PL, vanilla GD gets the linear rate $O(\log(1/\epsilon))$ even without convexity.
Loss landscape facts: for sufficiently wide neural networks, all local minima are global, and SGD’s noise gives implicit bias toward flat minima with better generalization.

Each is rigorous in its setting. The article also discusses what is not known — there is no general theorem saying “SGD finds the global optimum of a deep network.”

Optimization (10): Stochastic Optimization and Variance Reduction

Tue, 27 Sep 2022 09:00:00 +0000

Stochastic gradient descent samples a single component gradient per step — far cheaper than full GD, but at what cost in convergence? Can we retain the cheap per-iteration cost while recovering the fast rate of deterministic methods? This article quantifies the tradeoff from a noise-budget perspective and derives the solution.

\min_x f(x) := \frac{1}{n} \sum_{i=1}^n f_i(x),

deterministic gradient descent costs $$O(n)$$ per step but converges in $O(\kappa \log(1/\epsilon))$ steps. Stochastic gradient descent (SGD) costs $$O(1)$$ per step but converges in $O(1/\epsilon^2)$ for convex problems and $O(\kappa^2 \log(1/\epsilon))$ for strongly convex ones. Which is faster depends on $$n$$ , $\kappa$ , and $\epsilon$ .

Optimization (9): Interior-Point Methods and Self-Concordant Barriers

Mon, 26 Sep 2022 09:00:00 +0000

In 1984 Karmarkar showed that LPs could be solved in polynomial time practically — not just theoretically (the ellipsoid method had achieved this on paper). His interior-point method stayed inside the feasible polytope and converged in $$O(n L)$$ iterations, far better than the simplex method’s exponential worst case. Within a decade, Nesterov & Nemirovski generalized this to all convex programming via the self-concordant barrier framework. The result — $O(\sqrt{n} \log(1/\epsilon))$ Newton iterations for an $$n$$ -dimensional problem — remains the gold standard for medium-scale convex optimization.

Optimization (8): Lagrangian Duality and KKT Conditions

Sat, 24 Sep 2022 09:00:00 +0000

The most consequential idea in constrained optimization is that constraints have prices. The Lagrangian transforms a constrained problem into an unconstrained one by attaching a non-negative multiplier to each inequality and a free multiplier to each equality. The resulting unconstrained problem may be easier (the SVM dual), or it may give a verifiable lower bound (the LP duality used to certify integer programs).

This article develops:

Weak duality: the dual is always a lower bound on the primal — no assumptions needed.
Strong duality: under Slater’s condition (or convexity + linear constraints), the gap is zero.
KKT conditions: primal stationarity + dual feasibility + complementary slackness, the practical optimality system.
Saddle-point characterization: the Lagrangian’s saddle point coincides with the optimal primal–dual pair.

Each result is proved or carefully cited. We close with the SVM example, where the dual cuts the problem dimension from $$d$$ (number of features) to $$n$$ (number of training points) — the original kernel-method magic.

Optimization (7): Second-Order Methods

Thu, 22 Sep 2022 09:00:00 +0000

First-order methods top out at $O(\sqrt{\kappa})$ iterations to reach $\epsilon$ -accuracy (article 05). Second-order methods break this barrier by using curvature: Newton’s method has quadratic local convergence — the number of correct digits doubles every iteration — and quasi-Newton methods retain most of this speed without computing the Hessian explicitly.

The cost is in the per-iteration work: Newton solves an $n \times n$ linear system per step ( $$O(n^3)$$ ), BFGS maintains an $n \times n$ matrix ( $$O(n^2)$$ per step + $$O(n^2)$$ memory), and L-BFGS uses only $$O(mn)$$ memory for a chosen history $$m$$ (typically 5–20).

Optimization (6): Composite Optimization and Proximal Methods

Wed, 21 Sep 2022 09:00:00 +0000

When your objective contains a non-smooth piece (sparse regularisation, total variation, an indicator of a constraint set) or a constraint that is hard to handle directly, “just do gradient descent” stalls — there is no gradient at the kink, or every step violates feasibility. The proximal operator is the engineered, beautiful workaround: think of each update as “take a step on the smooth part, then run a tiny penalised minimisation that pulls the iterate back toward a structured solution”.

Optimization (5): Acceleration Beyond Nesterov

Tue, 20 Sep 2022 09:00:00 +0000

Article 02 introduced Nesterov acceleration and showed it improves the per-iteration cost from $\kappa$ to $\sqrt{\kappa}$ . This article asks the deeper questions:

Why $\sqrt{\kappa}$ and not faster? We prove a matching lower bound — no first-order method can do better.
Is Nesterov the only way? Polyak’s Heavy-Ball method achieves the same rate using a completely different update rule.
Can we accelerate any solver? The Catalyst framework wraps a black-box optimizer to gain the accelerated rate, at the cost of solving a regularized subproblem.

The unifying tool is a Lyapunov potential — a non-negative quantity that the algorithm decreases at every step. Both Nesterov and Heavy-Ball admit Lyapunov proofs, and the lower bound essentially says no Lyapunov decrease can happen faster.

Optimization (4): Learning Rate and Schedules

Sun, 18 Sep 2022 09:00:00 +0000

Your model diverges. You halve the learning rate. Now it trains, but takes forever. You halve again — now the loss is a flat line. Sound familiar? Of all the knobs you can turn, learning rate is the one that most often decides whether training converges, crawls, or blows up. This guide gives you the intuition, the minimal math, and a practical workflow to get it right — from a 12-layer CNN on your laptop to a 70B-parameter LLM on a thousand GPUs.

Optimization (3): The Gradient Descent Family from SGD to AdamW

Fri, 16 Sep 2022 09:00:00 +0000

Why is “tuning the LR is an art” a meme for ResNet, while every modern LLM paper just writes “AdamW, $\beta_1{=}0.9, \beta_2{=}0.95, \mathrm{wd}{=}0.1$ ” and moves on? It is not an accident — it is the end-point of three decades of optimizer evolution.

This post walks the lineage end-to-end on a single thread: each step exists because of a specific failure of the previous one. We end with the three directions that have actually entered the post-2023 large-model toolkit: Lion, Sophia, and Schedule-Free.

Optimization (2): Smoothness, Strong Convexity, and Nesterov Acceleration

Thu, 15 Sep 2022 09:00:00 +0000

A surprising amount of “optimizer folklore” collapses into three concepts:

How steep is the gradient? Lipschitz smoothness ( $$L$$ -smoothness) caps the step size.
How sharp is the bottom? $\mu$ -strong convexity sets the convergence rate and forces the minimizer to be unique.
Can we get there faster without losing stability? Nesterov acceleration and adaptive restart turn the per-condition-number cost from $\kappa$ into $\sqrt{\kappa}$ .

This post lays them out on a single thread: nail the geometric intuition with the minimum number of inequalities, prove the key theorems, then close with a least-squares experiment that pits GD, Heavy Ball, and Nesterov against each other. The goal is not to stack formulas — it is to make you able to look at a new problem and instantly answer “what step size, what rate, is acceleration worth it?”

Optimization (1): Convex Analysis Foundations

Wed, 14 Sep 2022 09:00:00 +0000

This article is the foundation the rest of the series is built on. Almost every result we will prove later — convergence rates of gradient descent, Lagrangian duality, the proximal operator, even the analysis of stochastic methods — relies on a small set of facts about convex sets and convex functions. We will derive all of them from scratch.

If you only remember three things from this article, make it these:

Kernel Methods (8): Deep Kernel Learning vs Deep Learning — A Practitioner's Guide

Thu, 30 Dec 2021 09:00:00 +0000

In 2026, why are you still reading about kernel methods? Aren’t transformers supposed to have eaten the entire ML stack? Yes and no. Transformers eat the headlines, but kernels still eat the corners — the regimes with 200 samples, the regimes where the model has to publish calibrated error bars, the regimes where a physicist needs to know which basis function caused the prediction. This final part is the field manual: when kernels actually win, how to debug them when they don’t, and how to bolt them on top of a neural network when you want the best of both worlds.

Kernel Methods (7): Large-Scale Kernels — Nystrom Approximation and Random Fourier Features

Fri, 24 Dec 2021 09:00:00 +0000

You want to train an RBF SVM on a million-image classification set. The Gram matrix is $10^6 \times 10^6$ doubles, which is 8 TB. That number alone — eight terabytes of RAM, just to store the kernel — is why most working data scientists who learned kernel methods in a stats class quietly never reach for them on real production workloads. The kernel trick gives you an infinite-dimensional feature space for the cost of one dot product per pair; the bill arrives when you have $$n^2$$ pairs.

Kernel Methods (6): Gaussian Processes — When Kernels Meet Bayesian Inference

Sun, 19 Dec 2021 09:00:00 +0000

Kernel ridge regression gives you a number. You feed it $$x_*$$ , it returns $\hat{y}_* = 23.7$ . End of story. But you wanted to act on that prediction — maybe schedule a delivery, dose a patient, place a bet — and the single number is not enough. Tomorrow’s temperature being “25°C” is useful; “very likely 25°C, 95% chance between 22 and 28” is actionable. Every decision under uncertainty needs the second one. Gaussian Processes are the cleanest way to upgrade a kernel method from “point predictor” to “distribution predictor”, and they do it without abandoning a single line of the kernel math from the previous five parts.

Kernel Methods (5): Kernel SVM, Kernel PCA, and Kernel Ridge Regression

Tue, 14 Dec 2021 09:00:00 +0000

Your features are two-dimensional, your data is clearly a circle inside a circle, and LinearSVC is at 50% accuracy with the wide-eyed look of an algorithm that genuinely believes a straight line is the answer. You stare at the scatter plot, you stare at the model, and somewhere in the back of your head the words kernel SVM surface. You type kernel='rbf', the accuracy jumps to 0.98, and the rest of the afternoon you wonder what exactly just happened — and why the same trick also gives you a Kernel PCA that unfolds a Swiss roll and a Kernel Ridge regressor that fits a sine wave with three lines of code.

Kernel Methods (4): Common Kernel Families — RBF, Matern, Polynomial, Periodic, and More

Thu, 09 Dec 2021 09:00:00 +0000

You type SVC(kernel='rbf') in scikit-learn for the first time. What did you set gamma to? 'scale'? 'auto'? You scrolled past those defaults without thinking. Three months later your model is overfitting, your Gram matrix looks like the identity, and you have no idea which knob is wrong. Most “kernel tuning” debt is really kernel choice debt — you picked the default kernel for the wrong reason, and now no amount of grid search will save you.

Kernel Methods (3): RKHS — The Theoretical Soul of Kernel Methods

Sat, 04 Dec 2021 09:00:00 +0000

If your eyes glaze over the moment a lecturer writes “RKHS” on the board, this part of the series is for you. RKHS is not a club of three intimidating letters — it is a function space, and once you see what lives inside it, kernel methods stop feeling like magic and start feeling like linear algebra you already know.

Kernel Methods (2): Mathematical Foundations — Positive-Definite Kernels and Mercer's Theorem

Mon, 29 Nov 2021 09:00:00 +0000

A week into kernel-SVM hacking I wrote what felt like a perfectly reasonable similarity function — tanh(1.5 * x.dot(y) - 2.0). It compiled, it ran, the math looked symmetric. Then sklearn coughed up ValueError: kernel matrix is not positive semidefinite and the optimiser produced a model that was worse than guessing.

That error message turned out to hide one of the deepest results in 20th-century analysis. “Positive-definite” is not a checkbox — it is the entire reason the kernel trick is allowed to exist. If your function is PSD, there exists a Hilbert space where it is a real inner product; if it is not, you are pretending to live in a space that nobody built. This post unpacks that statement, builds the operational tests, derives Mercer’s theorem, and works through enough numerical examples that the next time you see the failure message you will know exactly which line of math your kernel violated.

Kernel Methods (1): Why We Need Them — Hitting the Ceiling of Linear Algorithms

Wed, 24 Nov 2021 09:00:00 +0000

The first time I tried to fit a logistic regression to a dataset of two interlocking spirals, I burned an afternoon tweaking the regularizer, swapping solvers, and rescaling features — convinced that I was doing something wrong. The accuracy hovered around 50%. That is the noise floor of a coin flip; my model was, in a very literal sense, learning nothing.

The model was not buggy. The data was simply not the kind of object a straight line can describe. No amount of C, class_weight, or tol was going to change that. Once you have seen this failure mode once, you start noticing it everywhere — in customer-churn data with non-monotone relationships, in image classification before deep learning, in any regression where the trend bends. A linear algorithm has a hard ceiling, and you only break through that ceiling by changing the kind of object the algorithm operates on.