Forem: Alex Becker

Distributions vs Releases: Why Python Packaging is Hard

Alex Becker — Tue, 30 Apr 2019 14:43:23 +0000

Most programming language's package ecosystems have two levels: each package has one or more release, which are distinguished by a version. Python has a third: each release has one or more distributions, which are the actual files you download to install a package. In most languages said file is synonymous with the release, but the "or more" is crucial in Python, because for most releases of most widely-used packages, there is in fact more than one distribution.

Why? Well, Python is special in that it treats C extensions as a first-class feature of the language, and tries to insulate package users from having to compile C extensions. This means that distributions need to contain binary code compiled from the C extensions—such distributions (in their modern iteration) are called binary wheels. But C extensions usually need to be compiled for a specific target Python version and operating system, so to have any sort of wide support you need multiple wheels. Furthermore, since the package author can't anticipate all Python versions and operating systems (some of which don't exist yet!), it's also important to include a source distribution, which the package user is responsible for compiling.

Despite this, users—and most tools—still think in terms of release versions rather than specific distributions. This can lead to surprising inconsistencies. For example, installing a package might take seconds on one machine (because there is a matching binary distribution) and minutes or even hours on another. Even if both machines find appropriate binary distributions to install, their hashes won't match, making it more difficult to detect MitM attacks. This is because tools like pip automatically determine the "most suitable" distribution for a release, preferring binary wheels when one is compatible with the given system—and the most specific binary wheel if multiple are—and otherwise falling back to the source distribution. Most other tools follow suit, if only by virtue of using pip under the hood.

The biggest problems crop up when a new distribution is published after you've already installed another distribution for that release. This is all but inevitable—PyPI only lets you upload distributions one at a time, creating a new release with the first upload that has a new version, so eventually someone is bound to download that first distribution before you upload the last one. It's made much more common by the practice of having buildbots build different distributions in parallel, with binary distributions generally taking significantly longer than source distributions. But even worse is when a package author goes back to add support for a new platform—or a new version of python—for a release months or years after the fact. When this happens:

Build systems which expect a certain hash for a given package suddenly break.
PyPI mirrors like PyDist don't know to look for the new distribution and get out of sync.
Systems which have previously installed a distribution for the release (say, your dev machine) won't get the new distribution, and might behave differently from systems that install it (say, your production servers).

There is no obvious way to fix these issues without significantly disrupting the ecosystem—although the PyPI maintainers are aware of the pain and discussing tool improvements. In the meantime, it is incumbent on heavy Python users and system administrators to understand how Python packages are distributed and how pip chooses distributions.

Announcing the PyDist CLI

Alex Becker — Wed, 10 Apr 2019 14:30:33 +0000

One of the selling points of PyDist is that it works with standard Python tools—you don't have to install anything to use it. This is great for keeping containerized deployments and CI infrastructure lean and fast. But sometimes you just want to do as little configuration as possible and not have to keep track of the different Python tools and their quirks. That's why I've created the PyDist CLI, which combines the functionality of pip and twine while minimizing the amount of configuration you need to do and smoothing out their rough edges.

All you need to do to publish to and install from PyDist is install the CLI with pip install pydist-cli and provide your API key when you first run it. You can then publish packages to PyDist with pydist publish and install packages from PyDist with pydist install. You can also use it with PyPI using the --public flag.

The PyDist CLI is much more opinionated than the standard Python tooling. The install command uses PyDist by default but has an automatic fallback to PyPI, making it more reliable than pip install. The publish command handles building distributions (both binary and source), checking the README and metadata formatting, uploading, and cleaning up. This is appropriate for most packages—and avoids several pitfalls associated with twine, such picking up artifacts in the build/ directory from prior releases.

Python's except quirk

Alex Becker — Mon, 11 Mar 2019 16:01:41 +0000

Let me show you my favorite Python quirk. What would you expect this python code to do?

e = 2.718
try:
    1/0
except ZeroDivisionError as e:
    pass
print(e)

If you come to Python from another programming language, you might expect that the except clause introduces a nested scope, so assigning to e in the clause does not effect the pre-existing e variable in the outer scope. However, in python control structures do not generally introduce nested scoped (comprehensions being the exception), so with more python experience you would probably expect this to print a ZeroDivisionError instance.

Actually, in the standard CPython implementation it prints nothing; instead, the last line raises a NameError. Is this a bug? Actually, it was quite intentional. If you look at the bytecode generated by the except clause, you see:

LOAD_CONST 0 (None)
STORE_NAME 1 (e)
DELETE_NAME 1

When control flow exits the except block, python deletes the name from scope. Why? Because the exception holds a reference to the current stack frame, which contains everything in scope. Since python is manages memory primary via reference count, this means that nothing in the current scope will be freed until the next round of garbage collection runs, if at all. The current behavior is a compromise between memory usage, ease of implementation, and cleanliness of the language. It is a bit of a wart, but I think it embodies a one of the things I love about Python: not letting purity get in the way of practicality.

But that only explains the DELETE_NAME instruction. Why does CPython set e to None if it's going to delete it immediately afterwards? Well, imagine you had the same thought as the CPython team, and decided to clean up the exception reference at the end of your except block:

try:
    1/0
except ZeroDivisionError as e:
    ...
    del e

At the end of your except block, CPython will try to delete the name e—which you already deleted! To get around this, CPython assigns e = None before deleting e to guarantee that e exists.

Growth of the Python Ecosystem

Alex Becker — Mon, 11 Mar 2019 03:07:26 +0000

One of the cool things about building a PyPI mirror is having so much data about the Python ecosystem at my fingertips. I decided to explore how the ecosystem has been evolving since PyPI was created in 2003. Most of my analysis starts at 2005, since that's when PyPI added the upload_time field.

The Python ecosystem has been steadily growing throughout this period. After the first few years of hyper growth as PyPI gained near-full adoption in the Python community, the number of packages actively developed each year—meaning they had at least one release or new distribution uploaded—has increased 28% to 48% every year.

As this graph shows, the majority (~66%) of actively maintained packages each year are new , and the majority of those do not continue to be maintained. However, there is still steady and robust growth in the number of packages maintained for more than 1 year. Growth in releases has been even stronger, at 31% to 59% every year, although it has slowed down somewhat. This means that packages are getting more releases on average, which is a decent proxy for becoming more mature and better-maintained.

The most surprising result I stumbled upon came from looking at the number of releases per package. Some of these I was expecting based on my personal experience upgrading dependencies frequently, such as AWS's botocore at #15. But the cryptocurrency trading library cctx immediately stood out like a sore thumb. With 4659 releases at time of writing, it has more than 3 times as many releases as any other package—despite being less than 2 years old! Its libraries.io page usually times out after 30 seconds when I try to load it. I am not sure whether this is excellent or terrible maintainership, but it is certainly impressive.

Another interesting thing to look at is how practices around distributing Python packages are changing. The biggest change was of course the release of Python 3. Binary wheels, introduced in 2012 and codified in PEP 427, are generally accepted as the best way to distribute Python packages. But adoption among package authors has taken time. Python Wheels tracks the adoption of wheels among PyPI's 360 most downloaded packages; at time of writing 82.5% of them include wheel distributions in their releases. The long tail of the other >60k packages is lagging a bit behind, but the percentage of all releases that include at least 1 wheel distribution is growing steadily and just crossed 50% in 2018.

Not every package will be distributed as a wheel; in particular
psycopg2 will soon stop publishing wheels due to conflicts between the bundled LibSSL and the system's pre-existing LibSSL. But very few packages have such a reason not to be distributed as wheels, so I expect the rate of adoption to stay strong until 90% or more—which this graph suggests will happen by the end of 2022.

Using non-PyPI Package Indices

Alex Becker — Fri, 08 Mar 2019 02:49:28 +0000

By default, Python tools like pip install packages from PyPI, the Python Package Index. However, most tools allow you to use any server that implements the same API as PyPI (usually but not always the Simple Repository API). There are two main reasons you might want to use an alternative package index:

An index that mirrors PyPI let you keep installing packages even if they are deleted from PyPI or PyPI goes offline.
You can upload packages to a private index instead of PyPI if you do not want to make them public.

How to configure your tools depends on whether you are using an alternative index in place of or in addition to PyPI. Mirrors can be used either way, while a private index is usually used in addition to PyPI. Note that PyDist or a self-hosted devpi instance can act as both a mirror and a private index.

Unfortunately, you need to configure each of your tools individually.

Configuring `pip`

There are two configuration options for pip that allow it to use alternative indices:

The index URL, which is where pip will initially check for packages. It defaults to https://pypi.org/simple.
The extra index URLs which are where pip will check for packages if they are not found in the index url. If multiple are provided, they are checked in order.

If you are using a mirror, you can either set it as the index to install all packages through the mirror, or you can set it as an extra index to only use it as a backup. If you are using a private mirror to compliment PyPI, it is tempting to use it as an extra index. However, if a package by the same name as one of your private packages is published on PyPI, this will cause pip to install that package instead. Thus it is safest to set the private index's URL as the index and use PyPI as an extra index.

You can configure pip in a number of different ways; which you should choose depends on what is easiest to set in your infrastructure and how broadly you want the configuration to apply.

Via the command line arguments --index-url and --extra-index-url
Via the environment variables PIP_INDEX_URL and PIP_EXTRA_INDEX_URL (note that this can only be set once, so you can only add one extra index this way)
Via the index-url and extra-index-url settings in a pip.conf file which can live:
- In the root of your virtual environment (e.g. ./venv)
- In your user config's pip subdirectory (usually ~/.config/pip on Linux)
- In your system-wide config directory (/etc on Linux)

Configuring `twine`

The standard tool to upload Python packages is twine. However, only some
rivate python indices support uploading with twine—to my knowledge only PyDist and Gemfury do among hosted solutions. Other indices will generally offer their own python packages for deployment.

There are two ways to configure twine to upload to a repository other than PyPI:

Via the command line --repository-url argument
Via a .pypirc file in your home directory

The .pypirc file should look like:

[distutils]
index-servers =
    pydist

[pydist]
repository: <index-url>
username: <username>
password: <password>

If you include multiple index-servers in .pypirc, you can pass the name you gave the index server (pydist in the example above) to the --repository flag when uploading. The .pypirc file is convenient because you will not be prompted for a username/password.

Configuring `pipenv`

Pipenv is probably the most popular dependency-locking tool for Python. Some guides suggest using the PIP_INDEX_URL and PIP_EXTRA_INDEX_URL environment variables to configure the pipenv like you would pip, but this is not handled correctly and the maintainer told me not to use it.

Instead, you can use the [[source]] section at the top of your Pipfile:

[[source]]
url = '<index-url>'
verify_ssl = true
name = 'pydist'

If you want to use multiple package indices, you can include multiple [[source]] sections—when pipenv finds packages it tries them in the order they are specified, or if you declare a package like mypackage = { version="*", index="pydist" } it will try the specified index first. However, pipenv's handling of multiple indices is currently buggy.

How "pip install" Works

Alex Becker — Tue, 05 Mar 2019 07:45:14 +0000

What happens when you run pip install <somepackage>? A lot more than you might think. Python's package ecosystem is quite complex.

First pip needs to decide which distribution of the package to install.
This is more complex for Python than many other languages, since each version (or release) of a Python package usually has multiple distributions. There are 7 different kinds of distributions, but the most common these days are source distributions and binary wheels. A source distribution is exactly what it says on the tin—the raw Python and potentially C extension code that the package developers wrote. A binary wheel is a more complex archive format, which can contain compiled C extension code. This is convenient for users, because compiling, say, numpy from source takes a long time (~4 minutes on my desktop), and it is hard for package authors to ensure that their source code will compile on other people's machines. But it comes at a price--the compiled code is specific to the architecture and often the OS it was compiled on, so most packages with C extensions will build multiple wheel distributions, and pip needs to decide which if any are suitable for your computer.

To find the distributions available, pip requests https://pypi.org/simple/<somepackage>, which is a simple HTML page full of links, where the text of the link is the filename of the distribution. The filenames encode the version, kind of distribution, and for binary wheels, the architecture and OS they are compatible with. This format is complex enough to be covered by two different PEPs:

The version scheme is covered by PEP 440.
Binary wheel filename compatibility tags are the subject of PEP 425.

To select a distribution, pip first determines which distributions are compatible with your system and implementation of python. For binary wheels, it parses the filenames according to PEP 425, extracting the python implementation, application binary interface, and platform. The python implementation can be something as broad as py2.py3 (meaning "any implementation of python 2.X or 3.X") or it can specify a python interpreter and major version, such as pp35 (meaning PyPy version 3.5). The application binary interface is essentially what version of CPython's C-API the C extension code is compatible with, if there is any. Interpreting the platform portion of the compatibility tag is more difficult. It can be relatively obvious, like win32 for 32-bit Windows, but I am usually installing manylinux1 wheels. Which Linux distributions are compatible with manylinux1 is a subject of heavy debate on the distutils mailing list. Luckily the process for source distributions is simpler—all source distributions are assumed to be compatible, at least at this step in the process.

Once pip has a list of compatible distributions, it sorts them by version, chooses the most recent version, and then chooses the "best" distribution for that version. It prefers binary wheels if there are any, and if they are multiple it chooses the one most specific to the install environment. These are just pip's default preferences though—they can be configured with options like --no-binary or --prefer-binary. The "best" distribution is either downloaded or installed from the local cache, which on Linux is usually located in ~/.cache/pip.

Determining the dependencies for this distribution is not simple either. In theory, one could just use the requires_dist value from https://pypi.org/pypi/<somepackage>/<version>/json. However, this relies on the package author uploading the correct metadata, and older packaging clients do not do so. So in practice pip (and anyone else who wants to know the dependencies of a package) have to download and inspect it.

For binary wheels, the dependencies are listed in a file called METADATA. But for source distributions the dependencies are effectively whatever gets installed when you execute their setup.py script with the install command. There's no way to know unless you try it, which is what pip does! Specifically, it leverages setuptools to run install up to the point where it knows what dependencies to install. However, this can be further complicated by the fact that running install might itself require dependencies. The standard way to specify this in a Python package is to pass a the setup_requires argument to setuptools.setup. By way of setuptools, pip will run setup.py just enough to discover setup_requires, install those dependencies, then go back and execute setup.py again. Naturally, this is madness and setup_requires should never be used.

Once pip has a list of requirements, it starts this whole process over again for each required package, taking into account any constraints on its version. It builds a whole tree of packages this way, until every dependency of every distribution it has found is already in the tree. This process breaks of course if there is a dependency cycle, but it will always terminate—after all, there are only finitely many python packages!

What happens though if one of the distributions pip finds violates the requirements of another, for example if it pip first finds idna version 2.5 but then finds a distribution requiring idna<=2.4? Well, it ignores the requirement and installs idna anyway! There is a longstanding issue open to add a true dependency resolver to pip, with lots of false starts and partial implementations, but none have ever quite made it in. This is of course in large part due to the complexity of determining the dependencies for a python package—it is very difficult to build an efficient dependency resolver when determining the dependencies of a single candidate requires downloading and executing potentially megabytes of code!

Next pip has to actually build and install the package. If it downloaded a source distribution, and the wheel package is installed, it will first build a binary wheel specifically for your machine out of the source. Then it needs to determine which library directory to install the package in—the system's, the user's, or a virtualenv's? This is controlled by sys.prefix, which in turn is controlled by pip's executable path and the PYTHONPATH and PYTHONHOME environment variables. Finally, it moves the wheel files into the appropriate library directory, and compiles the python source files into bytecode for faster execution.

Now your package is installed! I've really only scratched the surface—there are dozens of options that change pip's behavior, many corner cases of other distribution types and platform limitations, and I didn't even touch on installing multiple packages (which is handled differently than a package with multiple dependencies). But I hope it this was informative, if not useful.