https://www.chenk.top/en/tags/self-supervised-learning/Recent content in Self-Supervised Learning on Chen Kai BlogHugoenWed, 07 May 2025 09:00:00 +0000https://www.chenk.top/en/transfer-learning/02-pre-training-and-fine-tuning/Wed, 07 May 2025 09:00:00 +0000https://www.chenk.top/en/transfer-learning/02-pre-training-and-fine-tuning/<p>BERT changed NLP overnight. A model pre-trained on Wikipedia and BookCorpus could be fine-tuned on a few thousand labelled examples and beat task-specific architectures that researchers had spent years hand-crafting. The same pattern repeated in vision (ImageNet pre-training, then SimCLR, MAE), in speech (wav2vec 2.0), and in code (Codex). Today, &ldquo;pre-train once, fine-tune everywhere&rdquo; is the default recipe of modern deep learning.</p> <p>But <em>why</em> does pre-training work? When should you freeze layers, when should you LoRA, and how small does your learning rate need to be? This article unpacks both the theory and the engineering practice behind the most successful transfer paradigm we have.</p>