Probability and Statistics (6): Estimation — MLE, MAP, and the Bias-Variance Story

Mon, 26 Aug 2024 09:00:00 +0000

Everything we’ve built so far — distributions, expectations, limit theorems — assumed we knew the parameters. The Gaussian has mean $\mu$ and variance $\sigma^2$ . The Binomial has $$n$$ trials with success probability $$p$$ . But in practice, you don’t know $\mu$ or $$p$$ . You observe data and try to figure them out.

This is estimation theory: the bridge between probability (where parameters are given) and statistics (where parameters are inferred). It’s also where the foundations of machine learning live. Every time you train a model, you are estimating parameters from data. The quality of that estimation determines whether your model generalizes or overfits.

Maximum Likelihood on Chen Kai Blog

Probability and Statistics (6): Estimation — MLE, MAP, and the Bias-Variance Story