Python Engineering on Chen Kai Blog

Python Engineering (8): Performance — Profiling, Caching, and Knowing When to Stop

Wed, 27 Apr 2022 09:00:00 +0000

Donald Knuth’s famous quote is often half-remembered. The full version is: “We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.” The second sentence is the key. Performance work isn’t about making everything fast; it’s about finding the 3% that matters and making that fast.

This article is about finding that 3%. You’ll learn to profile first, optimize second, and measure the impact of each change.

Python Engineering (7): Packaging — From pip install to PyPI

Sun, 24 Apr 2022 09:00:00 +0000

You wrote a useful utility. A colleague asks you to share it. You zip the folder and email it. They unzip it, run python main.py, and get ModuleNotFoundError because they do not have the dependencies. Then they install the dependencies, but the wrong versions. Then they have Python 3.8 and your f-string walrus operators do not parse.

Proper packaging eliminates all of this. With pip install your-tool, everything just works: correct dependencies, correct versions, and a clean CLI command.

Python Engineering (6): Concurrency — Threads, Processes, and asyncio

Thu, 21 Apr 2022 09:00:00 +0000

Your script downloads 100 files one at a time. Each download takes 2 seconds, mostly waiting for the server to respond. Total time: 200 seconds. Your CPU is idle for 99% of that time, wasting compute and money on network latency. Concurrency can fix this.

Python has three concurrency models, each designed for different problems. Choosing the wrong one can make your code slow or full of race conditions. This article explains when to use each.

Python Engineering (5): I/O, Serialization, and Data Formats

Tue, 19 Apr 2022 09:00:00 +0000

Most programs are just plumbing between data formats. Read a CSV, transform it, write JSON. Load a config file, validate it, pass settings to the application. Every Python developer writes this code, and most of them get encoding, path handling, or serialization subtleties wrong at least once.

This article covers every common I/O pattern in Python, from basic file reading to columnar data formats, with a focus on the pitfalls that waste your time.

Python Engineering (4): Type Hints, Linting, and Code Quality

Sun, 17 Apr 2022 09:00:00 +0000

Code reviews should be about logic and design, not about whether someone used single quotes or double quotes. Formatting debates are a waste of engineering time. The solution is to let machines handle style and let humans focus on correctness.

This article covers three layers of automated code quality: type hints catch logical errors before runtime, linters catch style violations and common bugs, and pre-commit hooks enforce everything automatically on every commit.

Python Engineering (3): Testing — pytest, Fixtures, and the Confidence Loop

Thu, 14 Apr 2022 09:00:00 +0000

You change one line and three unrelated features break. You refactor a function and spend two hours manually clicking through the app to check if everything still works. You deploy on Friday and get paged at midnight. All of these are symptoms of the same disease: no tests.

Tests are not bureaucracy. They are the fastest way to know that your code does what you think it does. A good test suite runs in seconds and catches the bugs that would take hours to find manually.

Python Engineering (2): Project Structure — From Script to Package

Tue, 12 Apr 2022 09:00:00 +0000

Every project starts as a single file. You write main.py, it works, you add features, and one day you realize you have 1,500 lines in one file with functions that call other functions that depend on globals defined 800 lines above. The code works, but nobody (including future you) can understand it.

The jump from script to package is the first real engineering decision in a Python project. Get it right early, and testing, packaging, and deployment become easier. Get it wrong, and you’ll spend weeks untangling circular imports.

Python Engineering (1): Environment Setup — pyenv, venv, and Dependency Hell

Sun, 10 Apr 2022 09:00:00 +0000

Every Python developer has lived through this moment: you run a script on your colleague’s machine and it crashes because they have Python 3.8 while you wrote it on 3.11. Or worse, you pip install something globally and break a completely unrelated project. Python’s environment story is powerful once you understand it, but the default experience is a minefield.

This article walks through the entire toolchain from scratch. By the end, you’ll have a reproducible, isolated, and version-pinned setup that works the same way on every machine.