https://www.chenk.top/en/tags/attention-sinks/Recent content in Attention-Sinks on Chen Kai BlogHugoenWed, 01 Apr 2026 09:00:00 +0000https://www.chenk.top/en/llm-engineering/06-long-context/Wed, 01 Apr 2026 09:00:00 +0000https://www.chenk.top/en/llm-engineering/06-long-context/<p>&ldquo;1M token context&rdquo; is one of the most over-claimed numbers in LLMs. A model can attend to 1M tokens — that&rsquo;s an architecture statement. A model can <em>use</em> information at position 800K to answer a question — that&rsquo;s a behavior statement, and it&rsquo;s more challenging. This chapter covers the math of position encoding, the engineering tricks that extend context beyond the training length, and why most long-context claims fail needle-in-a-haystack tests.</p>