Using pip-tools for local development with Docker

Oleh Korkh — Wed, 17 Mar 2021 16:40:51 +0000

Docker is a great tool even for local development. At the same time, managing Python requirements in containers can be tricky. The pretty standard behavior is to use requirements.txt file that adds to the image during the building process. Usually, we are using something like that in the Dockerfile:

...
COPY ./requirements /requirements.txt
RUN pip install -U pip && pip install --no-cache-dir  -r /requirements.txt && rm -rf /requirements.txt
...

These instructions work fine for production deployment but not very useful for the development stage. Periodically you need to install a new library. With using a virtual environment, without Docker, we could use pip install ... and then pip freeze > requirements.txt for storing the installed libraries' information to the file. But with Docker, it is not a case because we need to install libraries on the build stage. At the same time, it is hard to change requirements.txt manually without breaking dependencies.

There are available Python dependency managers, like Pipenv or Poetry. But I prefer to use pip-tools because it is more straightforward. We can add it to the container image by improving the following line:

RUN pip install -U pip pip-tools && pip install --no-cache-dir  -r /requirements.txt && rm -rf /requirements.txt

The use case of the tool is simple. First of all, we prepare a file with the name requirements.in that will contain a list of dependencies. Each line in this file is a dependency. We can also use the same rules for managing versions like the regular pip install command. For example:

django==3.1.*
celery
pillow

We can now use the command pip-compile requirements.in for generating requirements.txt with all necessary dependencies. With pip-tools installed in the container, we can use the command like that:

docker run -rm ... pip-compile requirements.in

Now, the file requirements.txt will be generated, including all required dependencies of described libraries. So, it is possible to rebuild the container for installing new libraries.

You can check the pip-tools documentation for available commands and options. For example, how to upgrade a particular library. It is a pretty simple yet powerful tool.

Forem: Oleh Korkh

Using pip-tools for local development with Docker