https://www.chenk.top/en/tags/flash-attention/Recent content in Flash Attention on Chen Kai BlogHugoenMon, 10 Nov 2025 09:00:00 +0000https://www.chenk.top/en/nlp/llm-architecture-deep-dive/Mon, 10 Nov 2025 09:00:00 +0000https://www.chenk.top/en/nlp/llm-architecture-deep-dive/<p>The 2017 Transformer paper drew one block. Every production LLM today still uses that diagram as a silhouette, but almost every internal piece has been replaced. Pre-norm replaced post-norm. RMSNorm replaced LayerNorm. SwiGLU replaced GELU. Rotary embeddings replaced sinusoids. Multi-head attention became grouped-query attention. The dense FFN sometimes became a sparse mixture of experts. And the inference loop is dominated by a data structure that doesn&rsquo;t appear in the original paper at all: the KV cache.</p>