Manage your Python Project End-to-End with PDM

Ryan Smith — Sun, 31 Dec 2023 17:43:40 +0000

Installing PDM
Starting a New Project
Setting up a Virtual Environment
- Virtual Environment Auto-creation
- Creating a Virtual Environment Yourself
- Using a Pre-existing Virtual Environment
Managing Dependencies
- Syncing Changes
- Adding Dependencies
- Removing Dependencies
- Updating Dependencies
PDM Scripts
Building and Publishing Distributions
A Note from the Author
References

PDM is a modern, PEP-compliant package and dependency manager, and a powerful development tool that makes managing your Python projects easier from starting to publishing.

Installing PDM

In order to install PDM you will need a Python environment >= 3.8.

Note: Most modern systems will probably have a system environment that meets this requirement, but if yours does not or if you prefer not to install anything in your system environment (even if it's just PDM) check out asdf or pyenv to help install and manage additional Python environments.

The PDM docs prescribe a few different ways to install PDM, but the ones I have found to be the easiest are these from the Other installation methods:

Using pip

# as a global site-package
pip install pdm

# or as a user-specific site-package
pip install --user pdm

As an asdf plugin
If you are already using asdf to manage additional Python environments (or if you just started to after reading the beginning of this section) there's also a PDM plugin for asdf:

asdf plugin add pdm
asdf install pdm latest

Starting a New Project

Initializing a new project is super easy with pdm init. Here I start a new project called "transient-fortitude":

# create my new project's root directory
mkdir transient-fortitude
cd transient-fortitude
# from my new project's root directory
pdm init

pdm init walks you through an interactive session and then initializes your project structure and metadata for you based on your answers to a series of questions like which interpreter you'd like to use for the project, and whether you'd like to create a virtual environment for the project.

Below is a sample of a session for transient-fortitude:

$ pdm init
Creating a pyproject.toml for PDM...
Please enter the Python interpreter to use
0. /home/avlwx/.asdf/installs/python/3.12.0/bin/python (3.12)
1. /home/avlwx/.asdf/installs/python/3.12.0/bin/python3.12 (3.12)
# ... many more, omitted for brevity ...
Please select (0): 0
Would you like to create a virtualenv with /home/avlwx/.asdf/installs/python/3.12.0/bin/python? [y/n] (y): y
Virtualenv is created successfully at /home/avlwx/repos/github/transient-fortitude/.venv
Project name (transient-fortitude): 
Project version (0.1.0): 
Is the project a library that is installable?
If yes, we will need to ask a few more questions to include the build backend [y/n] (n): n
License(SPDX name) (MIT): 
Author name (Ryan Smith): 
Author email (<default from git user.email>): 
Python requires('*' to allow any) (==3.12.*): 
Project is initialized successfully

Note: For the questions with no answer I elected to take the defaults, which PDM displays in parentheses at the end of a given prompt.

Here is the layout of my new project after running through pdm init (note: I shortened the full output here for brevity):

$ tree -aL 2 -I '__pycache__'
.
├── .gitignore      # 1
├── .pdm-python     # 2
├── pyproject.toml  # 3
├── README.md       # 4
├── src             # 5
│   └── transient_fortitude
├── tests           # 6
│   └── __init__.py
└── .venv           # 7

To summarize, PDM created a project in the src layout style including:

A boilerplate .gitignore that I've found works for many Python projects. You can further customize this .gitignore to your needs.
Some PDM metadata for tracking a project's current Python interpreter. As a developer you rarely have to worry about this.
My project's pyproject.toml.
The beginning of a project README.
My project source code tree including an empty top-level package. The top-level package name is derived from the project name, and is normalized to a valid Python package name.
An empty tests package.
A new virtual environment for my project.

Note: PDM can also create new projects from templates, but that is left as an exercise for the reader. For more information on creating a project from a template check out Create Project From a Template in the PDM docs.

Now that I have my initial project structure, I can set up my project for version control:

# from the project root directory
git init -b main
git add *  # normally should be used with caution, but is helpful here
git commit -m "Initial project structure"

And that's it! My new project structure is set up and ready to go.

Setting Up a Virtual Environment

In Starting a New Project I talked about how to initialize a project using pdm init and showed that by answering the prompt "Would you like to create a virtualenv with..." with "y" PDM creates a new virtual environment for me in my project's root directory.

While this is great for brand new projects it is still generally helpful to know how to use PDM to create new virtual environments in situations where you don't need to start from pdm init.

Virtual Environment Auto-creation

Let's say you've just cloned an existing PDM-managed project that you don't have a local virtual environment for yet. In this case the first time you run pdm install (or pdm sync) in that project's root directory PDM will automatically create a new virtual environment for that project in <project root>/.venv and provision it with that project's dependencies. The interpreter that is used by default depends on what your default global Python interpreter is.

Creating a Virtual Environment Yourself

Let's say that you want some extra control over the creation a project's virtual environment before provisioning it.

In this case you can create the virtual environment yourself, and you have several options to do so depending on your needs.

Unnamed, In-project Virtual Environment
This is automatically created in <project root>/.venv by running:

pdm venv create

Named Virtual Environment
You can also give your new virtual environment a name with the --name flag:

pdm venv create --name transient-fortitude

The location where new named virtual environments are managed is controlled by the PDM configuration setting venv.location. You can use pdm config to find out the value of this setting.

With Pip Pre-installed
To have pip pre-installed in your project virtual environment without having to add it as a dependency run:

pdm venv create --with-pip

You can configure PDM to automatically add pip to your new virtual environments by default and avoid using the --with-pip flag everytime by updating the PDM config setting venv.with_pip like so:

pdm config 'venv.with_pip' 'true'

As a PyCharm user I have found that having pip in my project virtual environment is helpful since PyCharm can have a hard time with project environments where setuptools is not installed.

Matching a Python Version Specifier
You can specify a Python version as an argument to pdm venv create and PDM will resolve a matching interpreter (if it can) and create your virtual environment with it.

# find a 3.11 interpreter
pdm venv create 3.11

# find specifically a 3.11.4 interpreter
pdm venv create 3.11.4

With a Specific Python Interpreter
Sometimes you want control over exactly which interpreter is used to create your virtual environment.

Instead of just a version specifier, you can instead specify an absolute path to a specific interpreter and PDM will create your virtual environment with it.

pdm venv create /path/to/python

For even more information on working with virtual environments check out Working with Virtual Environments in the PDM docs.

Using a Pre-existing Virtual Environment

If you're used to creating virtual environments by hand using venv or virtualenv directly and prefer to do this (or if it's muscle memory to the point that you do it without thinking) that's okay! You don't have to go back and remove, then recreate your environment with PDM in order to use PDM for your project.

You can tell PDM which virtual environment to use for your project by running:

# -i flag to ignore whatever interpreter PDM might already remember
# -f flag to tell PDM to use the first matching interpreter
pdm use -if /path/to/venv/bin/python
# ... proceed to provisioning, etc. ...

Managing Dependencies

In Starting a New Project and Setting up a Virtual Environment I showed various ways to create a virtual environment for a project.

Once you have a virtual environment setup you will want to add, remove, and update your project's dependencies as the needs of your project evolve.

Syncing Changes

When collaborating on a project with other developers it will be necessary to synchronize your local virtual environment with remote changes to the project's dependency set maintained in your project's pdm.lock file.

To synchronize your working set with your project's lockfile, run:

pdm sync --clean

This will ensure any dependencies that have been added or upgraded get added or upgraded, while any dependencies that have been removed are removed.

Note: without the --clean option PDM will not purge your working set of dependencies that have been removed from your project's lockfile.

Adding Dependencies

To add new dependencies, run:

pdm add PACKAGE [PACKAGE ...]

# to allow pre-release versions use the option --pre/--prerelease
pdm add --pre PACKAGE [PACKAGE ...]

By default, pdm add will add your new dependencies to the pyproject.toml, perform a lock operation, and finally sync your working set by installing or updating any packages resolved during the lock. This may result in packages already in your project's lockfile being updated unexpectedly. If you'd like to try and prevent unexpected updates you can use the --update-reuse flag to tell PDM to reuse the versions of packages already pinned in the lockfile where possible.

Removing Dependencies

To remove dependencies run:

pdm remove PACKAGE [PACKAGE ...]

Updating Dependencies

To update your dependencies, run:

# update all dependencies, including transitive dependencies
pdm update --update-all

# update just specific packages
pdm update --update-reuse PACKAGE [PACKAGE ...]

# update specific packages and their dependency trees
pdm update --update-eager PACKAGE [PACKAGE ...]

PDM Scripts

PDM provides a mechanism to execute arbitrary and user-defined scripts in an environment that is aware of the packages in your project.

Just pass your command and its options to pdm run and PDM will execute that program for you in a dependency-aware environment without having to activate your virtual environment. For those who are familiar with npm run, docker run, and podman run the concept is similar.

This allows you to do things like:

Run your application entrypoint scripts without having to activate your virtual environment.
Tinker in a Python interpreter where all of your project code and its dependencies are loaded.
Define scripts that can be used both locally and in CI to execute tests or run static analysis tools against your code consistently for both developers and CI systems.

The one I find to be most powerful, though, is this:

Define scripts that can be used both locally and in CI to execute tests or run static analysis tools against your code consistently for both developers and CI systems.

For example, you can add tools like bandit (security scanning), black (formatting), flake8 (linting), isort (import management), and mypy (type-checking) as development dependencies for your project, then define scripts in your pyproject.toml file under the table [tool.pdm.scripts] (see example below) to run these tools against your code both locally and in CI!

# example user-defined PDM Scripts for developer and CI tools
[tool.pdm.scripts]
check-formatting = { composite = [
    "isort --check --settings-path ./pyproject.toml src/ tests/",
    "black --check src/ tests/"
]}
format = { composite = [
    "isort --settings-path ./pyproject.toml src/ tests/",
    "black src/ tests/"
]}
lint = "flake8 --config .flake8 src/ tests/"
type-check = "mypy --config-file ./pyproject.toml src/"
security-scan = "bandit -rc pyproject.toml src/ tests/"

Using these scripts developers can run pdm format against their working copy before committing to version control so that it's counterpart pdm check-formatting doesn't fail in CI.

As another example developers can run pdm type-check against their working copy before committing to version control, allowing them time to preemptively fix any issues with their type annotations so there are no surprises in CI.

The ability to standardize both CI and developer tools like this prevents issues with flaky CI, or drift between CI and developer tools! No more "it passed on my machine, but fails in CI" and project maintainers don't have to update developer tools in multiple places.

Note: I first discussed using pdm run in order to execute scripts in a package-aware environment, and later omitted run in my examples. For user-defined scripts like I showed above PDM provides a convenient shortcut for users that allows you to execute either pdm run format or pdm format with the same result.

For more information on PDM Scripts check out PDM Scripts in the PDM docs.

Building and Publishing Distributions

Once my project is in a state where I feel comfortable publishing it to PyPI for all the world to use I can use PDM to help me handle both building and publishing my distribution.

To build your distribution(s), run:

# builds both a source and binary distribution
pdm build

# builds just a source distribution
pdm build --no-bdist

# builds just a binary distribution
pdm build --no-sdist

By default distributions are built under a directory called dist/. If you want to customize the output directory for your distributions use the -d flag:

# build under a directory called "build"
pdm build -d build

When you're ready to publish the artifacts you just built to PyPI, just run:

pdm publish --no-build

Note: If you plan to use pdm publish --no-build with your distributions from pdm build you should note that pdm publish expects the distribution artifacts to be under dist/, and as of PDM 2.11.1 this cannot be overridden. Because of this, the -d flag for pdm build is primarily useful for scratch builds, or builds that are inputs to some middleware that processes artifacts further before publishing them.

If you want to build and publish both a source and binary distribution to PyPI (all the default stuff), you can condense the steps above in to one command:

pdm publish

Publishing to a Private Registry

If instead you want to publish to a private registry, you can use the --repository option, along with --username and --password to do so:

pdm publish --no-build \
            --repository <your private registry URL> \
            --username <your registry username> \
            --password <your registry password>

Warning: Make sure any command tracing for your shell is turned off (e.g. set +x in bash) otherwise you could leak your private registry password to your terminal!

For more information on adding registries and repositories for your project, and maintaining registry and repository secrets as local PDM configuration see Configure the repositories for upload in the PDM docs.

A Note from the Author

This post is informed by my personal experience using PDM for managing my Python projects and is not intended to act as a complete reference for PDM's CLI. If you come across something I haven't mentioned here, it's not because it's not important it's only because I haven't come across it (yet).

For PDM's complete CLI reference check out CLI Reference in the PDM docs.

Additional References

In a few places I talk about PDM's configuration settings, but never dive in to that in detail. For more information on configuring PDM check out Configurations in the PDM docs.
This post is primarily informed by my personal experience using PDM for managing my Python projects. However, some of the language I use is informed by the PDM docs themselves to ensure consistency. A big shout out to the PDM Project for such an amazing tool!

Forem: Ryan Smith