Managing your Python project with a Makefile

Python projects are nowadays quite straight-forward. If you use VSCode or PyCharm, the complete setup and handling of the project is taken care of. Still, I tend to keep a Makefile around, because

1. I grew up using the terminal and somehow cannot let go
2. Sooner or later you want to integrate parts of your deployment into a CI/CD pipeline

CI/CD pipelines

Now, for point 2 you could argue: isn’t it sufficient if I just dump the respective steps as YAML in the CI/CD pipeline config? I would argue: NO! Having a lot of code in the pipeline config makes it

• difficult to migrate the code if you ever want to switch to a different platform
• easy to debug each step
• possible to kick off Makefile targets from other scripts or systems

This approach leads to:

1. lean CI/CD pipeline config
2. fat Makefiles, helper scripts and Dockerfiles

Makefile Recap

I kind of assume that you know a bit how Makefiles work, but just in case a quick summary:

Pointers to the official reference documentation

Quick Reference | String Manipulation | File Name Manipulation | The Two Flavors of Variables

Virtual Environment and Makefile

Makefiles don’t work like shell scripts. You have to run venv activate command before executing the actual command (in case you wonder what ?= means, check here):

VENV?=.venv
VENV_ACTIVATE=. $(VENV)/bin/activate
lint:
	$(VENV_ACTIVATE) && ruff src tests

If you use the python executabe or pip from the virtual environment, you don’t need to use $(VENV_ACTIVATE):

PIP=$(VENV)/bin/pip
install:
	$(PIP) install '.'

I define a VENV variable to allow overwriting the virtual environment via the command line:

make VENV=/some/other/venv lint

or

export VENV=/some/other/venv
make lint

setuptools with pyproject.toml

Here is my default template. It contains steps that I found useful over the years.

.PHONY: lint fmt install install-dev install-all test tags build clean help docs publish init
.DEFAULT_GOAL := help

VENV?=.venv
PIP=$(VENV)/bin/pip
PY=$(VENV)/bin/python
VENV_ACTIVATE=. $(VENV)/bin/activate

lint: ## lint the source code
	$(VENV_ACTIVATE) && ruff check src/ tests/
	$(VENV_ACTIVATE) && ruff format --check --exclude "src/*/_version.py" src/ tests/

fmt: ## format the source code with ruff
	$(VENV_ACTIVATE) && ruff format src/ tests/
	$(VENV_ACTIVATE) && ruff check --fix src/ tests/

install: ## install into current env
	$(PIP) install '.'

install-dev: ## install with dev dependencies and with editable flag
	$(PIP) install -e '.[dev]'

install-all: ## install with all dependencies (pyspark, airflow)
	$(PIP) install '.[all]'

test: ## run tests
	$(VENV_ACTIVATE) && pytest -vvs tests/

tags: ## build a ctags file for jwb's crappy editor
	ctags --languages=python -f tags -R src tests

build: clean ## build the package
	python -m build

clean: ## clean build artifacts and __pycache__ files up
	rm -rf dist/ build/ *.egg-info src/*.egg-info docs/_build
	find . -type f -name '*.py[co]' -delete -o -type d -name __pycache__ -delete
	# (optional) cd docs && $(MAKE) clean

help: ## this help
	@awk 'BEGIN {FS = ":.*?## "} /^[a-zA-Z0-9._-]+:.*?## / {printf "\033[1m\033[36m%-38s\033[0m %s\n", $$1, $$2}' $(MAKEFILE_LIST)

init: $(VENV)/init

$(VENV)/init: ## init the virtual environment
	python3 -m venv $(VENV)
	touch $@

The help target is special. It parses the Makefile and prints a short help (the text behind ##). I don’t remember exactly, but I think I took it from this post.

Sphinx-based docs

When working with Sphinx-based documentation:

# (optional) if you have Sphinx-based docs
docs:
	cd docs && $(MAKE) html

Publish to pypi

When you want to publish your package:

# (optional) if you want to publish your package
dist: clean build ## build the package distribution using twine
	$(PY) -m twine check dist/*

publish: dist ## publish the package to pypi
	$(PY) -m twine upload dist/*

When working with requirements.txt

# if you work with requirements.txt
deps: $(VENV)/init $(VENV)/requirements

$(VENV)/requirements: requirements.txt $(VENV)/init ## install requirements
	$(PIP) install -r $<
	touch $@

The deps (short for dependencies) target requires the file $(VENV)/requirements. This means that if $(VENV)/requirements is newer than the time make deps is invoked, the $(VENV)/requirements target will be executed.

The same is happening in this rule:

$(VENV)/requirements: requirements.txt $(VENV)/init

1. The output is $(VENV)/requirements (that is what the touch $@ does, it changes the timestamp of the target file).
2. If requirements.txt or $(VENV)/init are changed, (re-)run this target.

Practially this means that every time you modify requirements.txt, you can run make deps and the requirements will be installed. And if you did not modify requirements.txt, make will do nothing.

Setting the PYTHONPATH in the Makefile (optional)

If you, for whatever reason, need to set the PYTHONPATH to the directory of the Makefile, you can use this snippet:

# same as `export PYTHONPATH="$PWD:$PYTHONPATH"`
# see also https://stackoverflow.com/a/18137056
mkfile_path := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
PYTHONPATH:=$(mkfile_path)..:$(PYTHONPATH)
export PYTHONPATH

I rarely use it. Usually I apply it to projects that already modify the PYTHONPATH variable.

Standardisation

I (personally) settled for these targets in Python projects

lint

Check the code using flake8 or ruff. Also check the formatting

fmt

Format the source code. If possible also run ruff check --fix

install

install the package using pip

test

run pytest

build (optional)

build the package

clean

clean all the files generated by the previous commands

dist

prepare the package for distribution

publish

publish the package on pypi

dev

if it is a webapp, start the server in dev mode

help

show a short help of all make targets

docs

generate the documentation (optional)

If I work on a project without Makefile, I just create it, but I do not add it to the version control. Then I can locally stick to my targets.

poetry-based projects

Sometimes I use Poetry for my Python projects, in this case my Makefile looks a bit like this:

.PHONY: all clean test

# existing VIRTUAL_ENV might mess with poetry, so make sure it is gone
VIRTUAL_ENV=
unexport VIRTUAL_ENV

help:
	@awk 'BEGIN {FS = ":.*?## "} /^[a-zA-Z0-9._-]+:.*?## / {printf "\033[1m\033[36m%-38s\033[0m %s\n", $$1, $$2}' $(MAKEFILE_LIST)

test: ## run pytest
	poetry run pytest -rA -vvs --log-level INFO

install: ## install dependencies and the package to poetry venv
	poetry install

lint: ## run mypy and flake8 to check the code
	poetry run mypy src
	poetry run flake8 src tests

fmt: ## run black to format the code
	poetry run black src tests

See also

• GitHub - sio/Makefile.venv: Seamlessly manage Python virtual environment with a Makefile
• Using Variables (GNU make) & Setting (GNU make)
• gnu make - What’s the difference between := and = in Makefile? - Stack Overflow