https://www.chenk.top/en/tags/sre/Recent content in SRE on Chen Kai BlogHugoenFri, 26 May 2023 09:00:00 +0000https://www.chenk.top/en/cloud-computing/operations-devops/Fri, 26 May 2023 09:00:00 +0000https://www.chenk.top/en/cloud-computing/operations-devops/<p><figure class="article-figure"> <img src="https://blog-pic-ck.oss-cn-beijing.aliyuncs.com/posts/en/cloud-computing/operations-devops/illustration_1.png" alt="Cloud Computing (7): Cloud Operations and DevOps Practices — Chapter overview" loading="lazy" decoding="async" class="content-image"> </figure> </p> <p>In 2017 GitLab lost six hours of database state. An engineer, exhausted, ran <code>rm -rf</code> on the wrong server during an incident. The backup procedures had silently been broken for months; nobody noticed because no one was restoring from backups. The lesson is not &ldquo;be careful with rm&rdquo;. The lesson is that operations is a <em>system</em> — tools, runbooks, monitoring, automation, and the rituals around them. When the system is healthy, no single tired engineer can take down production. When the system is rotten, every late-night fix is one keystroke from disaster.</p>