Python Engineering (1): Environment Setup — pyenv, venv, and Dependency Hell

Every Python developer has lived through this moment: you run a script on your colleague’s machine and it crashes because they have Python 3.8 while you wrote it on 3.11. Or worse, you pip install something globally and break a completely unrelated project. Python’s environment story is powerful once you understand it, but the default experience is a minefield.

This article walks through the entire toolchain from scratch. By the end, you’ll have a reproducible, isolated, and version-pinned setup that works the same way on every machine.

The Python Version Problem#

Most operating systems come with a system Python. On macOS, it was Python 2.7 (removed in Monterey). On Ubuntu 22.04, it’s Python 3.10. This system Python is used by OS-level tools, and installing or upgrading packages in it can break your operating system.

The core problems:

Problem	Example
System Python is outdated	Ubuntu 20.04 ships 3.8, you need 3.11 features
Multiple projects need different versions	Project A needs 3.9, Project B needs 3.12
Global pip installs cause conflicts	Package X needs `requests>=2.28`, Package Y pins `requests==2.25`
Reproducibility fails	“Works on my machine” because versions differ
OS tools depend on system Python	`apt` on Ubuntu uses system Python internally

The solution is a three-layer stack:

pyenv manages Python versions (install 3.9, 3.10, and 3.11 side by side)
venv isolates per-project dependencies
pip-tools or Poetry pin exact versions for reproducibility

pyenv: Multiple Python Versions Without Pain#

pyenv intercepts the python command and redirects it to whichever version you have configured. It does this by inserting shims into your $PATH.

Installation#

On macOS:

1
brew install pyenv

On Linux:

1
curl https://pyenv.run | bash

After installation, add these lines to your shell config (~/.bashrc, ~/.zshrc, or ~/.bash_profile):

1
2
3
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"

Reload your shell:

1
source ~/.zshrc  # or ~/.bashrc

Verify the installation:

1
2
$ pyenv --version
pyenv 2.3.36

Installing Python Versions#

List available versions:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
$ pyenv install --list | grep "^  3\." | tail -10
  3.11.5
  3.11.6
  3.11.7
  3.12.0
  3.12.1
  3.12.2
  3.12.3
  3.12.4
  3.13.0a3
  3.13.0a4

Install a specific version:

1
2
3
4
$ pyenv install 3.11.7
Downloading Python-3.11.7.tar.xz...
Installing Python-3.11.7...
Installed Python-3.11.7 to /home/user/.pyenv/versions/3.11.7

On macOS, if the build fails, you likely need:

1
brew install openssl readline sqlite3 xz zlib tcl-tk

On Ubuntu/Debian:

1
2
3
4
sudo apt install -y make build-essential libssl-dev zlib1g-dev \
  libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm \
  libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev \
  libffi-dev liblzma-dev

Setting the Active Version#

pyenv has three levels of version selection, from most specific to least:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# Global default (lowest priority)
$ pyenv global 3.11.7

# Per-directory (creates .python-version file)
$ cd ~/projects/my-api
$ pyenv local 3.11.7
$ cat .python-version
3.11.7

# Current shell session only (highest priority)
$ pyenv shell 3.12.2

The resolution order is: shell > local (.python-version) > global (~/.pyenv/version).

Check which version is active and why:

1
2
3
4
5
6
7
8
$ pyenv version
3.11.7 (set by /home/user/projects/my-api/.python-version)

$ pyenv versions
  system
  3.9.18
* 3.11.7 (set by /home/user/projects/my-api/.python-version)
  3.12.2

Commit .python-version to your repo. This ensures every developer uses the same Python version. It costs nothing and prevents version-mismatch bugs.

Virtual Environments: Isolating Dependencies#

Dependency hell tangled wires vs clean resolved dependencies

Even with the right Python version, you still need dependency isolation. Without it, pip install puts packages into a shared location, and two projects needing different versions of the same package will conflict.

Creating a Virtual Environment#

1
2
$ cd ~/projects/my-api
$ python -m venv .venv

This creates a .venv directory containing:

1
2
3
4
5
.venv/
  bin/          # python, pip, activate scripts
  include/      # C headers for building extensions
  lib/          # installed packages go here
  pyvirst.cfg   # points back to the base Python

Activation#

1
2
3
4
5
6
7
8
9
# macOS / Linux
$ source .venv/bin/activate
(.venv) $

# Windows (PowerShell)
> .venv\Scripts\Activate.ps1

# Windows (cmd)
> .venv\Scripts\activate.bat

When activated, python and pip point to the venv copies:

1
2
3
4
5
(.venv) $ which python
/home/user/projects/my-api/.venv/bin/python

(.venv) $ which pip
/home/user/projects/my-api/.venv/bin/pip

Deactivate when done:

1
2
(.venv) $ deactivate
$

Why .venv?#

The . prefix hides it in file listings. Most tools (VS Code, PyCharm, pytest) auto-detect .venv. Add it to .gitignore immediately:

1
echo ".venv/" >> .gitignore

Never commit the virtual environment, as it contains platform-specific binaries and is not portable.

venv vs virtualenv vs conda#

Feature	venv	virtualenv	conda
Included in stdlib	Yes (3.3+)	No (pip install)	No (separate installer)
Speed	Moderate	Fast	Slow
Python version management	No	No	Yes
Non-Python dependencies	No	No	Yes (C libs, etc.)
Cross-platform	Yes	Yes	Yes
Environment size	Small	Small	Large (200MB+)
Best for	General Python	Legacy/speed	Data science with C deps

Recommendation: Use venv for most projects. Use conda only if you need compiled scientific libraries (CUDA, MKL) that are painful to build from source.

pip: The Package Installer#

Python virtual environment isolated bubbles each with differ

With your venv activated, pip installs packages into the isolated environment.

Basic Commands#

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
# Install a package
(.venv) $ pip install requests

# Install a specific version
(.venv) $ pip install requests==2.31.0

# Install with version constraints
(.venv) $ pip install "requests>=2.28,<3.0"

# Upgrade a package
(.venv) $ pip install --upgrade requests

# Uninstall
(.venv) $ pip uninstall requests

# Show package info
(.venv) $ pip show requests
Name: requests
Version: 2.31.0
Location: /home/user/projects/my-api/.venv/lib/python3.11/site-packages
Requires: certifi, charset-normalizer, idna, urllib3

requirements.txt#

The traditional way to record dependencies:

1
2
3
4
5
# Generate from current environment
(.venv) $ pip freeze > requirements.txt

# Install from file
(.venv) $ pip install -r requirements.txt

A typical requirements.txt from pip freeze:

1
2
3
4
5
certifi==2023.11.17
charset-normalizer==3.3.2
idna==3.6
requests==2.31.0
urllib3==2.1.0

The problem with pip freeze: it dumps every installed package, including transitive dependencies. You cannot tell which packages you actually need versus which are dependencies of dependencies. Removing a package leaves its dependencies behind.

pip-tools: Reproducible Installs#

pip-tools solves the pip freeze problem by separating what you want from what gets installed.

Installation#

1
(.venv) $ pip install pip-tools

Workflow#

Create requirements.in with your direct dependencies:

1
2
3
4
# requirements.in
requests>=2.28
flask>=3.0
pydantic>=2.0

Compile it to a fully pinned requirements.txt:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
(.venv) $ pip-compile requirements.in
#
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
#    pip-compile requirements.in
#
blinker==1.7.0
    # via flask
certifi==2023.11.17
    # via requests
charset-normalizer==3.3.2
    # via requests
click==8.1.7
    # via flask
flask==3.0.0
    # via -r requirements.in
idna==3.6
    # via requests
itsdangerous==2.1.2
    # via flask
jinja2==3.1.2
    # via flask
markupsafe==2.1.3
    # via
    #   jinja2
    #   werkzeug
pydantic==2.5.3
    # via -r requirements.in
pydantic-core==2.14.6
    # via pydantic
requests==2.31.0
    # via -r requirements.in
urllib3==2.1.0
    # via requests
werkzeug==3.0.1
    # via flask

Every line shows where each dependency comes from. Sync your environment exactly:

1
(.venv) $ pip-sync requirements.txt

pip-sync removes packages not in the file, unlike pip install -r which only adds.

Upgrading#

1
2
3
4
5
# Upgrade all packages
(.venv) $ pip-compile --upgrade requirements.in

# Upgrade one package
(.venv) $ pip-compile --upgrade-package requests requirements.in

Dev Dependencies#

Create a separate file for development tools:

1
2
3
4
5
6
# requirements-dev.in
-c requirements.txt
pytest>=7.0
pytest-cov
mypy
ruff

The -c requirements.txt constrains dev deps to be compatible with production deps.

1
2
(.venv) $ pip-compile requirements-dev.in
(.venv) $ pip-sync requirements.txt requirements-dev.txt

uv: The Modern Alternative (2024+)#

uv is a Rust-based Python package manager that replaces pip, pip-tools, virtualenv, and pyenv in a single binary. It is 10-100x faster than pip and handles the entire workflow.

Installation#

1
2
3
4
5
6
7
8
# macOS / Linux
curl -LsSf https://astral.sh/uv/install.sh | sh

# macOS (Homebrew)
brew install uv

# Windows
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"

Python Version Management (replaces pyenv)#

1
2
3
4
5
6
7
8
9
# Install Python versions
$ uv python install 3.11 3.12 3.13

# Pin version for a project
$ uv python pin 3.12
Pinned `.python-version` to `3.12`

# List installed versions
$ uv python list

uv downloads pre-built Python binaries — no compilation needed, unlike pyenv which builds from source.

Project Initialization#

1
2
3
4
5
6
7
# Create a new project with pyproject.toml
$ uv init my-api
$ cd my-api

# Or initialize in an existing directory
$ cd existing-project
$ uv init

This creates a pyproject.toml with PEP 621 metadata and a uv.lock lockfile.

Dependency Management#

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
# Add a dependency (auto-creates venv, resolves, locks, installs)
$ uv add requests flask "pydantic>=2.0"

# Add dev dependency
$ uv add --dev pytest ruff mypy

# Remove a dependency
$ uv remove flask

# Sync environment to match lockfile exactly
$ uv sync

# Upgrade a specific package
$ uv lock --upgrade-package requests
$ uv sync

uv add does in one command what previously required editing requirements.in, running pip-compile, and running pip-sync.

Running Commands#

1
2
3
4
5
6
7
8
# Run a script in the project environment (auto-syncs first)
$ uv run python main.py
$ uv run pytest
$ uv run ruff check .

# Run a tool without installing it permanently
$ uvx ruff check .
$ uvx black .

Migration from pip-tools#

1
2
3
4
5
# Import existing requirements.in
$ uv add $(cat requirements.in | grep -v "^#" | grep -v "^-")

# Or start fresh from pyproject.toml
# uv reads [project.dependencies] directly

uv.lock vs requirements.txt#

Aspect	requirements.txt (pip-tools)	uv.lock
Format	Plain text, pip-compatible	TOML, uv-specific
Cross-platform	Single platform	Multi-platform resolution
Hash verification	Optional (`--generate-hashes`)	Always included
Speed	Seconds	Milliseconds
Resolution	Platform-specific	Universal (resolves for all platforms at once)

The uv.lock file resolves dependencies for all platforms simultaneously. A Linux developer and a macOS developer using the same lockfile get compatible (but potentially different) packages for their platform — automatically.

When to Use uv vs pip-tools#

Scenario	Recommendation
New projects (2024+)	uv — faster, simpler, fewer tools to manage
Legacy projects with established CI	pip-tools — proven, no migration risk
Libraries published to PyPI	uv or Poetry — both handle build+publish
Corporate environments with strict tool approval	pip-tools — zero external dependencies beyond pip
Docker builds where layer caching matters	uv — deterministic, fast installs

CI/CD: Verifying Your Dependency Lock#

A lockfile only works if it stays in sync with your declared dependencies. CI should verify this automatically.

GitHub Actions: Lock File Freshness Check#

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# .github/workflows/deps.yml
name: Dependency Check

on:
  pull_request:
    paths:
      - "pyproject.toml"
      - "requirements*.in"
      - "uv.lock"
      - "requirements*.txt"

jobs:
  check-lock:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      # Option A: uv
      - uses: astral-sh/setup-uv@v4
      - run: uv lock --check  # Fails if uv.lock is outdated

      # Option B: pip-tools
      # - run: pip install pip-tools
      # - run: pip-compile --quiet requirements.in
      # - run: git diff --exit-code requirements.txt

uv lock --check exits with code 1 if uv.lock would change — meaning someone edited pyproject.toml but forgot to update the lockfile.

Dependabot / Renovate Integration#

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# .github/dependabot.yml (for pip-tools)
version: 2
updates:
  - package-ecosystem: "pip"
    directory: "/"
    schedule:
      interval: "weekly"
    open-pull-requests-limit: 5

# For uv: Renovate supports uv.lock natively since v39

Security Auditing in CI#

1
2
# Check for known vulnerabilities in locked dependencies
- run: uv pip audit  # or: pip-audit -r requirements.txt

Docker: Reproducible Builds with Pinned Dependencies#

Docker images should produce identical results regardless of when or where they are built. This requires careful handling of Python dependencies.

Multi-Stage Build Pattern#

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
# Stage 1: Build dependencies (with build tools)
FROM python:3.12-slim AS builder

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir --prefix=/install -r requirements.txt

# Stage 2: Runtime (minimal image)
FROM python:3.12-slim AS runtime

COPY --from=builder /install /usr/local
COPY src/ /app/src/

WORKDIR /app
USER nobody
CMD ["python", "-m", "my_api"]

Docker + uv (Faster Builds)#

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
FROM python:3.12-slim

# Install uv
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv

WORKDIR /app
COPY pyproject.toml uv.lock ./

# Install dependencies (cached layer if lockfile unchanged)
RUN uv sync --frozen --no-dev --no-editable

COPY src/ ./src/
USER nobody
CMD ["uv", "run", "python", "-m", "my_api"]

--frozen refuses to update the lockfile — if it is outdated, the build fails rather than silently installing different versions.

Layer Caching Strategy#

1
2
3
4
5
6
7
8
# BAD: any source change invalidates dependency cache
COPY . .
RUN pip install -r requirements.txt

# GOOD: dependencies cached unless lockfile changes
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .

Copy dependency files first, install, then copy source. Docker reuses the cached dependency layer as long as requirements.txt (or uv.lock) hasn’t changed.

Environment Variables and .env Files#

Production applications often need runtime configuration (API keys, database URLs). Never hardcode these.

python-dotenv#

1
(.venv) $ pip install python-dotenv

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# settings.py
from pathlib import Path
from dotenv import load_dotenv
import os

load_dotenv()  # Reads .env file in project root

DATABASE_URL = os.environ["DATABASE_URL"]
API_KEY = os.environ["API_KEY"]
DEBUG = os.environ.get("DEBUG", "false").lower() == "true"

1
2
3
4
# .env (gitignored!)
DATABASE_URL=postgresql://user:pass@localhost:5432/mydb
API_KEY=sk-development-key-here
DEBUG=true

1
2
3
4
# .env.example (committed — shows required variables without real values)
DATABASE_URL=postgresql://user:pass@host:5432/dbname
API_KEY=your-api-key-here
DEBUG=false

Pydantic Settings (Type-Safe Configuration)#

For larger applications, use Pydantic to validate environment variables at startup:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
from pydantic_settings import BaseSettings

class Settings(BaseSettings):
    database_url: str
    api_key: str
    debug: bool = False
    max_connections: int = 10
    allowed_origins: list[str] = ["http://localhost:3000"]

    class Config:
        env_file = ".env"

# Fails fast at startup if required vars are missing
settings = Settings()

This gives you type validation, default values, and clear error messages when configuration is wrong.

Poetry vs pip-tools vs PDM#

Feature	pip-tools	Poetry	PDM
Config file	requirements.in	pyproject.toml	pyproject.toml
Lock file	requirements.txt	poetry.lock	pdm.lock
Venv management	No (manual)	Yes (auto)	Yes (auto)
Build & publish	No	Yes	Yes
Speed	Fast	Moderate	Fast
PEP 621 compliant	N/A	No (custom format)	Yes
Learning curve	Low	Medium	Medium
Stability	Very stable	Stable	Stable
Resolver	pip’s resolver	Custom	Custom

pip-tools is the simplest choice: it stays close to pip and adds only what is needed. Poetry is popular for libraries and applications that need build+publish. PDM follows PEP standards most closely.

pyproject.toml: The Modern Standard#

pyproject.toml replaces setup.py, setup.cfg, MANIFEST.in, and most tool-specific config files. It is defined by PEP 518 and PEP 621.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
[build-system]
requires = ["setuptools>=68.0", "wheel"]
build-backend = "setuptools.backends._legacy:_Backend"

[project]
name = "my-api"
version = "0.1.0"
description = "A sample API project"
readme = "README.md"
requires-python = ">=3.11"
license = {text = "MIT"}
authors = [
    {name = "Your Name", email = "you@example.com"},
]
dependencies = [
    "requests>=2.28",
    "flask>=3.0",
    "pydantic>=2.0",
]

[project.optional-dependencies]
dev = [
    "pytest>=7.0",
    "pytest-cov",
    "mypy",
    "ruff",
]

[project.scripts]
my-api = "my_api.cli:main"

[tool.ruff]
line-length = 88
target-version = "py311"

[tool.mypy]
python_version = "3.11"
strict = true

[tool.pytest.ini_options]
testpaths = ["tests"]

All tool configuration in one file. No more scattered .flake8, mypy.ini, pytest.ini.

Real Workflow: Clone to Running#

Here is what a correct setup looks like from zero:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# 1. Clone the repo
$ git clone git@github.com:team/my-api.git
$ cd my-api

# 2. pyenv reads .python-version automatically
$ python --version
Python 3.11.7

# 3. Create and activate venv
$ python -m venv .venv
$ source .venv/bin/activate

# 4. Install dependencies
(.venv) $ pip install -r requirements.txt
# Or with pip-tools:
(.venv) $ pip-sync requirements.txt requirements-dev.txt

# 5. Verify
(.venv) $ python -m pytest
========================= test session starts ==========================
collected 42 items
...
========================= 42 passed in 3.21s ===========================

# 6. Run the application
(.venv) $ python -m my_api
 * Running on http://127.0.0.1:5000

Automate steps 2-4 with a Makefile:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
.PHONY: setup test run

setup:
	python -m venv .venv
	.venv/bin/pip install --upgrade pip
	.venv/bin/pip-sync requirements.txt requirements-dev.txt

test:
	.venv/bin/python -m pytest

run:
	.venv/bin/python -m my_api

Common Pitfalls#

Pitfall	Symptom	Solution
Using system pip	`pip install X` modifies system packages	Always activate venv first
Forgetting to pin versions	`pip install requests` installs latest	Use `pip-compile` to pin
Committing .venv	Huge repo, platform-specific binaries	Add `.venv/` to `.gitignore`
pyenv not in PATH	`pyenv: command not found`	Add init lines to shell config
Build deps missing on Linux	`ModuleNotFoundError` during `pyenv install`	Install build-essential, libssl-dev, etc.
Conflicting global/local Python	Wrong version active	Check `pyenv version` to see which config wins
pip cache stale	Old package version installed despite upgrade	`pip install --no-cache-dir` or `pip cache purge`
Mixing conda and pip	Broken environment state	Pick one: conda OR venv+pip
requirements.txt has no hashes	Supply chain attack risk	`pip-compile --generate-hashes`
Forgot to update lock file	New dev gets different versions	CI should verify lock file is up to date

Directory Structure After Setup#

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
my-api/
  .python-version        # pyenv reads this (committed)
  .venv/                  # virtual environment (gitignored)
  pyproject.toml          # project metadata and tool config
  requirements.in         # direct dependencies
  requirements.txt        # pinned full dependency tree
  requirements-dev.in     # dev-only direct dependencies
  requirements-dev.txt    # pinned dev dependency tree
  .gitignore              # includes .venv/
  src/
    my_api/
      __init__.py
      ...
  tests/
    ...

What’s Next#

With your environment locked down, the next question is how to organize your code. A single main.py works for scripts, but anything beyond 200 lines needs structure. In the next article, we will build a proper Python project from scratch, covering package layouts, imports, entry points, and CLI tools.

The Python Version Problem#

pyenv: Multiple Python Versions Without Pain#

Installation#

Installing Python Versions#

Setting the Active Version#

Virtual Environments: Isolating Dependencies#

Creating a Virtual Environment#

Activation#

Why .venv?#

venv vs virtualenv vs conda#

pip: The Package Installer#

Basic Commands#

requirements.txt#

pip-tools: Reproducible Installs#

Installation#

Workflow#

Upgrading#

Dev Dependencies#

uv: The Modern Alternative (2024+)#

Installation#

Python Version Management (replaces pyenv)#

Project Initialization#

Dependency Management#

Running Commands#

Migration from pip-tools#

uv.lock vs requirements.txt#

When to Use uv vs pip-tools#

CI/CD: Verifying Your Dependency Lock#

GitHub Actions: Lock File Freshness Check#

Dependabot / Renovate Integration#

Security Auditing in CI#

Docker: Reproducible Builds with Pinned Dependencies#

Multi-Stage Build Pattern#

Docker + uv (Faster Builds)#

Layer Caching Strategy#

Environment Variables and .env Files#

python-dotenv#

Pydantic Settings (Type-Safe Configuration)#

Poetry vs pip-tools vs PDM#

pyproject.toml: The Modern Standard#

Real Workflow: Clone to Running#

Common Pitfalls#

Directory Structure After Setup#

What’s Next#

Python Engineering 8 parts

Liked this piece?