https://www.chenk.top/en/tags/pretraining/Recent content in Pretraining on Chen Kai BlogHugoenSun, 29 Mar 2026 09:00:00 +0000https://www.chenk.top/en/llm-engineering/03-pretraining/Sun, 29 Mar 2026 09:00:00 +0000https://www.chenk.top/en/llm-engineering/03-pretraining/<p>Pretraining is where most of an LLM&rsquo;s capability comes from, and it&rsquo;s also where the leaderboard-vs-reality gap is widest. Most published runs are heroic engineering more than they are scientific results. This chapter is about the parts of pretraining that you actually have to get right when you&rsquo;re not OpenAI: the data, the parallelism choice, and the failure modes that only show up when the cluster is large enough to make a single bad NCCL all-reduce kill a 30-day run.</p>