Forem: Bruno Alla

Migrating a project to Poetry

Bruno Alla — Mon, 19 Oct 2020 13:30:00 +0000

Poetry is a tool solving the problem of Python packaging. It was started back in February 2018 by Sébastien Eustace (also the author of pendulum). It has a beautiful website and a ambitious headline:

Python packaging and dependency management made easy

It has been on my radar for a while, but I never gave it a proper go. I was happily using pip-tools, which was solving my main use case, while being a lot more lightweight and meant I could keep working with pip since its output is a good old requirements.txt. I heard about it a few times online, often next to Pipenv, but recently it looks like Poetry got a bit more traction.

Having a bit of time on my hands, a few weeks ago I decided to take a proper look at it and maybe migrate one of my projects, Deezer Python.

Starting point

Before the migration, here is how the package was managed:

Project metadata were in setup.cfg, using setuptools declarative config.
Development (and documentation) dependencies managed by pip-tools.
Documentation hosted on Read the Docs (RTD).
Releases automated with Python Semantic Release (PSR) on Github Actions.
Development tools configured in setup.cfg (black, isort, pyupgrade, flake8).
Using tox to help local testing.
Using Github Actions for CI.

These are a couple of features that were impacted by migrating to Poetry, and I think it’s worth mentioning them for context. Since Poetry uses pyproject.toml, I was also hoping to move all my tools config from setup.cfg to that file.

Migration

Installing Poetry

The first step is to get the CLI. I initially installed it via Homebrew, but later realised that Poetry was setting some default values based on the Python version its installation uses. As the Homebrew Python can be updated without notice, I realised it was not the best option here, so I later reinstalled it via pipx using a Python installation managed by pyenv that wouldn’t be wiped without my knowledge:

pipx install \
 --python ~/.pyenv/versions/3.8.6/bin/python \
 poetry

Project metadata

The first step was to migrate the project metadata from setup.cfg to pyproject.toml. Poetry comes with a handy interactive command poetry init which will create a minimal pyproject.toml for you. I already noticed a few pleasant surprises:

The CLI was very nice to interact with
The author and author_email from setup.cfg were merge into an array of authors, each with the format Full Name <email@address.com>.

I then went on to convert more settings into the new format manually, and the process was quite painless. Many settings have the same name and values, and when they are different, it’s mainly to simplify things. I guess it’s something a library like setuptools cannot easily afford to do due to backwards compatibility, but that a new opt-in tool like Poetry can.

Dependencies

Adding dependencies and development dependencies was pretty simple, I just needed to run poetry add [-D] ... with the list of packages at the end.

In this process, I discovered that one of the development dependency, pyupgrade is not compatible with Python 3.6.0: Poetry would not let me set my own Python to ^3.6. I initially changed my minimum version to ^3.6.1 as a quick fix, but I realised later that it impacted my package's trove classifiers, the ones generated by Poetry: my package wasn't listed as Python 3.6 compatible. Since this is just for a development dependency, there is a better way! The fix is to specify the dependency conditional to the Python version:

[tool.poetry.dependencies]
python = "^3.6"
...

[tool.poetry.dev-dependencies]
...
pyupgrade = { version = "^2.7.3", python = "^3.6.1" }

With this, the minimum Python version of my package and its classifiers are correct.

Extra Dependencies

This package had an “extra require” dependency. There is a good example in the documentation, these dependencies needs to be specified in the same section as normal dependencies, but as optional:

[tool.poetry.dependencies]
...
tornado = {version = "^6.0.4", optional = true}

And a dedicated pyproject.toml section maps the extras to an array of optional dependencies:

[tool.poetry.extras]
tornado = ["tornado"]

I got confused initially because I thought it was done via the poetry add ... -E ... command. However, it does something different, it’s for the extra of the dependency, not the extra of my own library the dependency should fall under.

For example Django has 2 possible extras at the moment, so one would run the following commands to use them:

poetry add Django -E argon2

It means “install Django with the argon2 extra”. I thought it meant “install Django and put it under the argon2 extra”.

Docs dependencies

The dependencies to build the docs were specified in a requirements.txt in the docs/ folder and RTD was configured to pick this up. I initially thought that I wouldn’t be able to remove that file, but it turns out it’s possible to make it work.

Thanks to PEP 517, which Poetry is compliant with, you can do pip install . in a Poetry package. This has been in pip since 19.0, and pip running on RTD is newer than this. However, this method wouldn’t install your development dependencies, so your docs dependencies cannot be specified as such. It works if you specify a docs extra, though:

# pyproject.toml
[tool.poetry.dependencies]
...
myst-parser = {version = "^0.12", optional = true}
sphinx = {version = "^3", optional = true}
sphinx-autobuild = {version = "^2020.9.1", optional = true}
sphinx-rtd-theme = {version = "^0.5", optional = true}

[tool.poetry.extras]
...
docs = [
    "myst-parser",
    "sphinx",
    "sphinx-autobuild",
    "sphinx-rtd-theme",
]

# readthedocs.yml
version: 2
python:
  install:
    - method: pip
      path: .
      extra_requirements:
        - docs

The downside of this, is that the extra is part of your package, so not ideal.

Releases

I recently moved the automation of releases to Python Semantic Release which worked well for me, and this would have been a blocker if it wouldn’t work. These are the pieces I needed:

Move its config to come from pyproject.toml.
Package version is specified in pyproject.toml as well as in __init__.py.
Update build command to use Poetry instead of setuptools.

Here is the PSR config in pyproject.toml to achieve that:

[tool.semantic_release]
version_variable = [
    "deezer/ __init__.py: __version__",
    "pyproject.toml:version"
]
build_command = "pip install poetry && poetry build"

I was expecting to have to change more, but it all worked out of the box with just that.

Linting and code formatting

All the tools I use for linting and code formatting were configured via setup.cfg and ideally I’d like to replace it by pyproject.toml. It was possible for almost everything, except for flake8 which has an open issue for it.

I decided to move as much things as I could to pyproject.toml, and move flake8 config to .flake8.

With all the above, I was able to remove the setup.cfg as well as all the pip-tools files for dependencies.

Code Coverage

Pytest had and unexpected config section for pyproject.toml, I didn't pay much attention to it initially and put its config under [tool.pytest], while the section should actually be [tool.pytest.ini_options].

The consequence of this was that my config was ignored and I didn't run the tests with coverage enabled, which silenced another error in the coverage section tool.coverage.run, where source should be an array:

[tool.pytest.ini_options]
addopts = "-v -Wdefault --cov=deezer"

[tool.coverage.run]
branch = true
source = ["deezer"]

With this, pytest was running with coverage, but for some reason, codecov wasn't picking up the report properly. I noticed that it used to find an xml file when it was working, so I edited the command on CI to generate it with --cov-report=xml:

poetry run pytest --cov-report=xml

With this changes I eventually got my code coverage reporting back, but it's something I missed in the original migration. Not directly related to Poetry, but interesting that Pytest decided to go with this section name.

Tox

Poetry works nicely with Tox, I followed the section in their FAQ, and here is a overview of the changes:

[tox]
isolated_build = true
envlist = py36,py37,py38,py39,pypy3,docs,lint,bandit

[testenv]
whitelist_externals = poetry
commands =
    poetry install
    poetry run pytest
...

I replaced the deps section in each testenv by a poetry install into the list of commands to run, and prefixed all commands to be run in the isolated environment by poetry run.

Github Actions

Poetry isn’t installed out of the box on Github Actions, one could either install it with a simple run step or use a dedicated action for it. I’ve opted for the dedicated action, thinking that Dependabot could keep it up to date for me.

The rest of the changes are pretty simple, it’s a matter or replacing pip install by poetry install -E ... and prefixing all commands by poetry run. My docs were tested and I was changing directory with cd, I took this opportunity to instead use working-directory key to the Github action step.

Verdict

Did Poetry deliver on its ambitious tagline? I think so, I was really impressed by the developer experience of Poetry, its CLI is really nice, and I hit little issues on the way. Overall the migration was not too difficult, you can check the pull request on Github. I feel like there are quite a few features I just scratched the surface (like multi-environments).

I’m going to wait a bit to see how this works in the longer run, but I think I’ll migrate my other projects soon.

Auto-update pre-commit hooks with GitHub Actions

Bruno Alla — Sun, 12 Jul 2020 00:00:00 +0000

Pre-commit hooks are great to reduce the feedback loop for things like linting and auto-formatting. Git supports them out of the box, but they are not easy to share across all developers working on a project, they need to be installed by each developers.

Several tools exist to solve this problem, but my favorite is pre-commit. It’s written in Python, but it aims at being language agnostic. It saves your setup in a config file and developers can install all of them with a single command.

Each tool is referenced by their github repo and tag to install, which is great because each tool is pinned to a specific version. However, I usually have the tool versions already elsewhere in my repository, for example in requirements.txt, causing some duplication. The main project dependencies are automatically updated with Dependabot, PyUP or Renovate but none of these tools supports the pre-commit config file. After a while, it's easy to end up with versions discrepancies.

That is until this week-end, where I stumbled upon the autoupdate command from pre-commit. I’m not sure how I missed this before, it looks like it’s been part of pre-commit for a really long time. By combining this with the power of Github actions, I was able to get it to send me a pull request each time a new version is available:

name: Pre-commit auto-update

on:
  schedule:
    - cron: '0 0 * * *'

jobs:
  auto-update:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2

      - name: Set up Python
        uses: actions/setup-python@v2
        with:
          python-version: 3.8

      - name: Install pre-commit
        run: pip install pre-commit

      - name: Run pre-commit autoupdate
        run: pre-commit autoupdate

      - name: Create Pull Request
        uses: peter-evans/create-pull-request@v2
        with:
          token: ${{ secrets.CPR_GITHUB_TOKEN }}
          branch: update/pre-commit-autoupdate
          title: Auto-update pre-commit hooks
          commit-message: Auto-update pre-commit hooks
          body: |
            Update versions of tools in pre-commit 
            configs to latest version
          labels: dependencies

This workflow is scheduled every day at midnight, runs pre-commit autoupdate and sends a pull request if there are any changes.

The piece that required a bit of fiddling is the action creating the pull request, partly to get commit message, title, content and labels right, but mostly because I initially used secrets.GITHUB_TOKEN as token, but it wouldn't trigger the CI build for that pull request.

It's a limitation which is well documented on the action's README, and is intentional from Github. I chose the solution to create a PAT scoped to repo and added it to the secrets as CPR_GITHUB_TOKEN. It's deployed and running on the repo of django-codemod.

The pull request action has fixed inputs, so it will create one pull request at a time for all updates. If several tools get a new version, they would all be updated at once, and if a pull request already exists, it would receive more updates. This is not necessarily a bad thing, but if one tool breaks the build due to new linting rules, all are stuck.

Maybe I'll look into making the pull request content a bit more dynamic, but for now it does the job I need to. I'm also planning to add this to my Cookiecutter template for Python package, so I can get it for all my new projects.

I hope this can help folks keep their pre-commit hook up to date, maybe this will become obsolete when pre-commit CI is ready, or maybe it will be a cheaper and simpler alternative

Static vs. Media files in Django

Bruno Alla — Tue, 18 Jun 2019 00:00:00 +0000

Django is a great Web framework for building Web applications in Python. It comes with a lot of batteries included that you’ll most likely need at some point in your project. Two of them took me a while to differentiate when I started: static and media files.

After helping some less experienced people, I feel like I’m not the only one running into the confusion, so I’m hoping to clarify their differences in this post.

TL;DR

Static files are part of your application code, while media is for generated content by your application or users of your application.

Static files

Static files are managed by the staticfiles app which you need to install. It's made of several building blocks, the 3 most important ones being:

Storage classes
Templates tags
collectstatic admin command

These components work together to serve the assets in a more or less optimised way, depending on the environment. This can be altered using the following settings:

STATIC_ROOT
STATIC_URL
STATICFILES_DIRS
STATICFILES_STORAGE

Static files are usually either part of your code, or part of your dependencies’ code. They can come from various places, each app may provide its own files. They are typically kept in source control. The Django admin ships with some javascript and CSS, for example, that are stored in Django's Github repository.

Local development setup

In development, the setup for static files is inefficient and optimised for convenience. It’s based on a view that, by default, looks into all the installed apps to find static files. Works great for local development, but not ideal for production.

Production setup

In production, finding files is done ahead of time via the collectstatic admin command, which you should run as part of your deployment. At a high level, the command does a very similar job as the development view, but the main difference is that it runs outside of the request-response cycle, without blocking the person visiting your website. It copies all the static files into a single location, being another folder or somewhere on another machine, which could be in a different part of the world.

My go-to solution for this used to be in django-storages. It ships with a storage class for saving your files into AWS S3. I would also add a CDN in front to help caching the assets closer to my users. The setup looked like this:

However, a couple of years ago I’ve switched to Whitenoise, which leads to a simpler setup, and remove the need for a S3 bucket, all hosting comes from the Django app. The CDN should serve most of the traffic anyway:

I still use django-storages but only for media files, which brings us to the next section…

Media files

Media files are usually for files which are uploaded by users or generated by your application during the life of your Django project. They are typically not stored in source control.

This could be in the admin, the profile picture the user set in thei profile, some more private documents or a PDF receipt for an order that your app generate.

Basically, anything which is going into a FileField, ImageField or the likes is classified as media storage.

By default, files are stored on the local file system, and again a few settings are here to configure the behaviour:

MEDIA_ROOT
MEDIA_URL
DEFAULT_FILE_STORAGE

In some production environments -if you run your app on Heroku for instance- this default setup is not suitable. Heroku has an ephemeral file system meaning that it's cleared each time the application is updated. This is where django-storages is still useful and needed.

If your app is hosting a mix of public and private medias, I recommend to define separate storage classes for each, you can tell in the field instantiation which storage class to use:

class User:
    name = models.CharField()
    # public file
    profile_picture = models.ImageField(
        storage=PublicStorage()
    )
    # private file
    dbs_check_document = models.FileField(
        storage=PrivateStorage()
    )

The django-storages library has several backends for various hosting providers with a lot of options to customise them, so I won’t go into the details of each, but here is an example of what it could look like for AWS:

from storages.backends.s3boto3 import S3Boto3Storage

class PublicStorage(S3Boto3Storage):
    default_acl = 'public'
    file_overwrite = False
    bucket_name = 'my-public-bucket'

class PrivateStorage(S3Boto3Storage):
    default_acl = 'private'
    file_overwrite = False
    bucket_name = 'my-private-bucket'

Use different class locally and in production

There is one problem with the above approach, though: by defining your storage class on the model field, that means it would potentially be shared between you local environment and production, which is probably not what you want. To solve this, we can use a factory function to either use the local file system or a remote location, depending on a setting:

def get_storage(storage_type):
    if settings.USE_REMOTE_STORAGE:
        storage_class = {
            'private': PrivateStorage,
            'public': PublicStorage,
        }[storage_type]
        return storage_class()
    return FileSystemStorage()

Which would change your model definition to:

class User:
    name = models.CharField()
    # public file
    profile_picture = models.ImageField(
        storage=get_storage('public')
    )
    # private file
    dbs_check_document = models.FileField(
        storage=get_storage('private')
    )

In a real world case, this helper should probably handle KeyError in the lookup and return the DEFAULT_FILE_STORAGE. You could also invent custom settings as you need.

Security concerns

If you decide to use django-storages for both Static and Media files, it’s important that you make the separation very clear, a user of your website shouldn’t be able to override one of your static assets by uploading a resource! I like to use a separate S3 Bucket for each type of storage. AWS has many options to customise the privacy of buckets and the objects their contain, and other providers have equivalent features.

Final note

While the Media storage might not be used by your application, the static files are a more essential, and any reasonable Django site will need it (at least for the admin).

I hope this post will help reduce the confusion I see frequently with beginners.