Functional Analysis on Chen Kai Blog

Kernel Methods (3): RKHS — The Theoretical Soul of Kernel Methods

Sat, 04 Dec 2021 09:00:00 +0000

If your eyes glaze over the moment a lecturer writes “RKHS” on the board, this part of the series is for you. RKHS is not a club of three intimidating letters — it is a function space, and once you see what lives inside it, kernel methods stop feeling like magic and start feeling like linear algebra you already know.

Kernel Methods (2): Mathematical Foundations — Positive-Definite Kernels and Mercer's Theorem

Mon, 29 Nov 2021 09:00:00 +0000

A week into kernel-SVM hacking I wrote what felt like a perfectly reasonable similarity function — tanh(1.5 * x.dot(y) - 2.0). It compiled, it ran, the math looked symmetric. Then sklearn coughed up ValueError: kernel matrix is not positive semidefinite and the optimiser produced a model that was worse than guessing.

That error message turned out to hide one of the deepest results in 20th-century analysis. “Positive-definite” is not a checkbox — it is the entire reason the kernel trick is allowed to exist. If your function is PSD, there exists a Hilbert space where it is a real inner product; if it is not, you are pretending to live in a space that nobody built. This post unpacks that statement, builds the operational tests, derives Mercer’s theorem, and works through enough numerical examples that the next time you see the failure message you will know exactly which line of math your kernel violated.

Functional Analysis (12): Functional Analysis in Action — PDE and Quantum Mechanics

Sat, 23 Oct 2021 09:00:00 +0000

Eleven articles is a long time to spend on infrastructure. Normed spaces, Banach and Hilbert structure, dual spaces, weak topologies, bounded and unbounded operators, the spectral theorem, semigroups, distributions, Sobolev spaces — every one of those chapters paid for itself with a clean abstract result, but a reader could be forgiven for wondering when the abstraction was going to do anything. This final article is where I make good on the implicit promise of the series: every theorem we built was built because some concrete problem demanded it, and pulling those threads together gives us the modern toolkit for partial differential equations and quantum mechanics.

Functional Analysis (11): Distributions and Sobolev Spaces — Generalized Solutions

Thu, 21 Oct 2021 09:00:00 +0000

I want to start with a confession. For years I treated the Dirac delta the way an undergraduate physicist does: as a function that is zero everywhere except at the origin, where it is infinite, and whose integral is one. That description is, of course, mathematical nonsense. No measurable function can have those properties. Yet every quantum mechanics textbook uses $\delta$ on page one, every signal processing course writes $\delta(t)$ for an impulse, and every PDE book invokes Green’s functions $$E$$ satisfying $\Delta E = \delta$ . Either an entire scientific community has been making a fundamental error for a century, or there is a way to make this object rigorous. The latter, obviously — and the way is the theory of distributions.

Functional Analysis (10): Semigroups of Operators — Evolution Equations in Infinite Dimensions

Tue, 19 Oct 2021 09:00:00 +0000

The simplest interesting differential equation is $$u' = a u$$ , with $a \in \mathbb{R}$ . The solution $u(t) = e^{at} u_0$ is so familiar that it is easy to forget it is a piece of structure: the map $T(t) = e^{at}$ is a one-parameter family of operators on $\mathbb{R}$ satisfying $$T(0) = I$$ , $$T(t + s) = T(t) T(s)$$ , and continuity in $$t$$ . Replace $$a$$ with a self-adjoint matrix $$A$$ and you have $T(t) = e^{tA}$ , the matrix exponential, which solves the system $$u' = Au$$ . Replace $$A$$ with an unbounded operator on a Hilbert space — the Laplacian, the Schrödinger Hamiltonian, a Fokker-Planck operator — and you would like to do the same thing. But the matrix-exponential power series may not converge, the operator may not be defined on all of $$H$$ , and ordinary calculus stops working.

Functional Analysis (9): Unbounded Operators — When Boundedness Fails

Sun, 17 Oct 2021 09:00:00 +0000

Two articles ago I was talking about how spectral theory is the linear-algebraic infrastructure of quantum mechanics. The trouble is that nearly every operator a physicist actually cares about – the position operator, the momentum operator, the Laplacian, the Schrodinger Hamiltonian – is not bounded. They are not defined on the whole Hilbert space. They are densely defined, with domains that depend on the regularity or decay of the input function. None of the previous spectral apparatus applies directly. We need to extend it.

Functional Analysis (8): Spectral Theory — Decomposing Operators

Fri, 15 Oct 2021 09:00:00 +0000

When I first saw the word “spectrum” used for an operator I assumed it was a fancy synonym for “set of eigenvalues.” That is the right intuition for matrices and for compact operators, and it is exactly what one wants in introductory linear algebra. The trouble is that it is wrong as soon as the operator is not compact. The position operator $$(Mf)(x) = x f(x)$$ on $$L^2[0, 1]$$ has no eigenvalues: any eigenfunction would have to satisfy $x f(x) = \lambda f(x)$ a.e., which forces $$f = 0$$ everywhere away from a single point, hence $$f = 0$$ in $$L^2$$ . And yet the operator is clearly not invertible, since $\lambda I - M$ is multiplication by $x - \lambda$ , which fails to be boundedly invertible whenever $\lambda \in [0, 1]$ .

Functional Analysis (7): Compact Operators — The Bridge to Finite Dimensions

Wed, 13 Oct 2021 09:00:00 +0000

I owe my fondness for compact operators to a small embarrassment. As an undergraduate I assumed that infinite-dimensional linear algebra would feel exotic everywhere. It does not. There is a wide and well-mapped suburb of operator theory in which everything one learned about symmetric matrices – eigenvalues, orthogonal eigenvectors, the spectral decomposition – comes back almost unchanged, just with eigenvalues tailing off to zero instead of a finite list. That suburb is the world of compact operators, and the price of admission is a single condition: the operator must squeeze the unit ball into a relatively compact set. Once that condition is met, nearly everything follows: the spectrum is countable, nonzero eigenvalues are isolated with finite-dimensional eigenspaces, the Fredholm alternative holds, and integral equations of the second kind become as tractable as linear systems. The line between matrices and infinite-dimensional operators ceases to be a wall and becomes a permeable membrane.

Functional Analysis (6): Bounded Linear Operators and the Big Theorems

Mon, 11 Oct 2021 09:00:00 +0000

Why This Article Is Where the Theory Catches Fire#

For five articles I have been building scaffolding: metric and normed spaces, Hilbert spaces, dual spaces, weak topologies. None of those individually felt very impressive — I am, after all, just doing topology and linear algebra in slightly more general settings than usual. The point at which functional analysis genuinely delivers is right here, in the three great theorems of Banach space operator theory: the Uniform Boundedness Principle, the Open Mapping Theorem, and the Closed Graph Theorem. Each of these takes a piece of “pointwise” or “set-theoretic” data — pointwise boundedness, surjectivity, closedness of the graph — and concludes a global structural property — uniform boundedness, openness, continuity — that has no analog in finite dimensions because finite-dimensional linear algebra makes them all true automatically.

Functional Analysis (5): Weak and Weak-* Topologies — When Norm Convergence Is Too Strong

Sat, 09 Oct 2021 09:00:00 +0000

Why Weaker Topologies Exist and Why They Matter#

Article 1 ended with a depressing fact: in any infinite-dimensional normed space, the closed unit ball is not compact. No bounded sequence is guaranteed to have a norm-convergent subsequence. If you are trying to find a minimizer of an energy functional — say, the lowest-energy configuration of a vibrating membrane — you take a minimizing sequence, and you need a limit. In finite dimensions, Bolzano-Weierstrass delivers that limit. In infinite dimensions, it does not. The direct method of the calculus of variations appears dead on arrival.

Functional Analysis (4): Dual Spaces and the Hahn-Banach Theorem — Taming Linear Functionals

Thu, 07 Oct 2021 09:00:00 +0000

Why You Cannot Skip This Article#

Up to now, the theory has been about spaces and the elements that live in them. This article changes the perspective: it asks what you can say about a vector $$x$$ by measuring $$x$$ against a family of test functionals. The shift from “vectors” to “vectors plus functionals” is what turns Banach spaces into a serviceable analogue of finite-dimensional linear algebra. In finite dimensions, every linear functional is continuous and the dual space is the same dimension as the original — so there is nothing to prove. In infinite dimensions, continuity is a real constraint, and the existence of enough continuous functionals to separate points or extend partial data is not obvious. The Hahn-Banach theorem is what guarantees this, and it is the result that makes functional analysis possible.

Functional Analysis (3): Hilbert Spaces — Geometry in Infinite Dimensions

Tue, 05 Oct 2021 09:00:00 +0000

Inner Products and the Geometry They Create#

If a Banach space is a normed space that has agreed to be complete, a Hilbert space is a Banach space that has further agreed to admit angles. That extra agreement — an inner product — is what restores almost all of finite-dimensional geometry to the infinite-dimensional setting. Orthogonality, projection, the Pythagorean theorem, the notion of “closest point in a subspace” — all come back unchanged. The price of admission is a single axiom; the reward, geometric and computational, is enormous.

Functional Analysis (2): Normed Spaces and Banach Spaces

Sun, 03 Oct 2021 09:00:00 +0000

Why a Norm Is More Than a Metric Wearing a Hat#

In Article 1, the metric was a free-standing function on a set with no algebraic structure. That generality bought us topology and completeness, but it gave nothing back to the algebra. The moment I am willing to assume the underlying set is a vector space, a more rigid object becomes available: a norm, a single nonnegative function on the space whose induced metric $d(x,y) = \|x - y\|$ is translation-invariant and positively homogeneous.

Functional Analysis (1): Metric Spaces — Distance, Convergence, and Completeness

Fri, 01 Oct 2021 09:00:00 +0000

Why I Had to Stop Trusting My Finite-Dimensional Intuition#

The first thing graduate analysis did to me was take away my picture. Up to that point, “distance” had always been the length of an arrow drawn from the origin to a point — Pythagoras, three coordinates, done. Then somebody asked me how far two functions are from each other and the arrow disappeared.