https://www.chenk.top/en/tags/information-entropy/Recent content in Information Entropy on Chen Kai BlogHugoenMon, 26 Jan 2026 09:00:00 +0000https://www.chenk.top/en/ml-math-derivations/07-decision-trees/Mon, 26 Jan 2026 09:00:00 +0000https://www.chenk.top/en/ml-math-derivations/07-decision-trees/<blockquote> <p><strong>Hook.</strong> A decision tree mimics how humans actually decide things: ask a question, branch on the answer, ask the next question. The math under that intuition is surprisingly rich — entropy from information theory tells us <em>which</em> question to ask first, the Gini index gives a cheaper proxy that lands on essentially the same trees, and cost-complexity pruning gives a principled way to stop the tree from memorising noise. Almost every modern boosted ensemble (XGBoost, LightGBM, CatBoost) is just a clever sum of these objects, so getting the foundations right pays off many times over.</p>