https://www.chenk.top/en/tags/tokenization/Recent content in Tokenization on Chen Kai BlogHugoenSat, 28 Mar 2026 09:00:00 +0000https://www.chenk.top/en/llm-engineering/02-tokenization/Sat, 28 Mar 2026 09:00:00 +0000https://www.chenk.top/en/llm-engineering/02-tokenization/<p>Tokenization is the layer everyone skips. It&rsquo;s also the layer where I&rsquo;ve debugged the most production bugs — silent quality regressions, mysterious cost spikes, models refusing to follow instructions because someone formatted the chat template wrong. This chapter is everything I wish I&rsquo;d internalized before shipping a multilingual product.</p> <p><figure class="article-figure"> <img src="https://blog-pic-ck.oss-cn-beijing.aliyuncs.com/posts/en/llm-engineering/02-tokenization/illustration_1.png" alt="LLM Engineering (2): Tokenization Deep Dive — Chapter overview" loading="lazy" decoding="async" class="content-image"> </figure> </p> <hr> <h2 id="what-a-tokenizer-actually-does" class="heading-anchor">What a tokenizer actually does<a href="#what-a-tokenizer-actually-does" class="heading-link" aria-label="Permalink to this section" title="Copy link to this section">#</a> </h2><p>A tokenizer maps a string to a list of integer IDs. Reverse maps IDs back to a string. Both directions are deterministic but not bijective in general — round-tripping <code>tokenizer.decode(tokenizer.encode(s))</code> can lose whitespace, normalize Unicode, or collapse repeated punctuation, depending on the algorithm.</p>