Forem: Fabien Arcellier

capture and stream the standard output of a process with python on linux and windows

Fabien Arcellier — Sun, 24 Sep 2023 16:00:00 +0000

Invoking and interacting with a program from a Python script may get complicated. In this article, I share solutions that could make your life easier. You should read till the end if you are working on code that has to run on both linux and windows.

alfred is an opensource tool that I develop. Its development confronted me with problems I had been trying to avoid for years. Handling standard and error outputs is hard. I was looking for :

display real-time standard output and error output for a better user experience
capture standard output and error to be able to process it at the end of execution
display real-time standard output and error output on both windows and linux
preserve the possibility of being able to interact with the terminal

In this article, I share answers to the first 3 problems. The last point is not complete. To go further, it requires me to explore the notion of pseudo terminal. I am looking for open source cross platform pty library that is working well on both linux and windows. If you know one, please, share it in the comments.

Handling standard outputs

In a program, I know 3 streams which serve as input and output. Standard input passes data entered by the user to a program. Standard output displays data to the user. Error output displays error data to the user.

Standard outputs are data streams. They are sequential. The parent process can only read them once. Process can access them through file descriptors. A subprocess can write to standard output and error output and it can read standard input.



import sys

sys.stdout.write("hello world")
sys.stderr.write("error message")
value = sys.stdin.read()

architecture of I/O run through popen (source)

Reading a file is a blocking operation in Python. Remember it, it will be important for the future. As long as there are no bytes and the stream remains open, the program will remain stuck on the instruction. There is no common API between Linux and Windows for non-blocking reading. It is this point which is the main problem to have a portable solution.

write a subroutine

We will work on Linux first, we will do the tests on Windows at the end of the article. A shell script will play the role of executable for the rest. You will need to assign execution rights to it with the chmod +x .suprogram command.

./subprogram



#!/bin/sh

echo "What is your name? "
read my_name

RED='\033[0;31m'
NC='\033[0m' # No Color
echo "Hello ${RED}${my_name}${NC}"

sleep 3
echo "hello world"

run this subroutine and let the parent shell handle standard output

The simplest case is to run a program and not interact with it from our script.



import subprocess

p = subprocess.Popen(["./suprogram"], stdout=None, stderr=None)
result_code = p.wait()
print("return to the python workflow")

The subroutine display the outputs. You can perform keyboard input . But, we can't retrieve what is on screen in our python script. If you don't need it, this is the best solution for running a subroutine.

run a program and capture standard output

We capture the standard output and leave the error type separate. The python script receive the standard outputs. The python script print what appears on standard output. Error output will stay invisible.



import subprocess
import sys

p = subprocess.Popen(["./subprogram"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)

stdout_content = []
while True:
    line = p.stdout.readline().decode("utf-8")
    stdout_content.append(line)
    sys.stdout.write(line)
    if line == '' and p.poll() != None:
        break

p.wait()
stdout_content = ''.join(stdout_content)
print("return to the python workflow")

I inspected the stdout_content variable in the pycharm debugger. It contains the contents of the standard output.

If we want to capture standard output and error output at the same time, this pattern will fail.

readings on standard output and error output block each other

Do you remember that the read() method is blocking? If we write to stdout, reading from stdout will be fine but reading from stderr will remain blocked. The rest of the contents of stdout will stay unvisible. The read on stderr remains blocked.

The situation may temporary evolve when the sub program writes data on stderr. But, as soon as the buffer is empty, the program will stay stuck et on the read of stderr again.



p = subprocess.Popen(["./subprogram"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)

while True:
    stdout_line = p.stdout.readline().decode("utf-8")
    stderr_line = p.stderr.readline().decode("utf-8") # BLOCKING
    # ...

    if line == '' and p.poll() != None:
        break

Python does not offer a mechanism for non-blocking reading on both Linux and Windows. I'll let you discover for yourself the solution using select that works on Linux.

capture both standard output and error output and display them in real time

The threads will allow us to capture and display the 2 flows in real time. I encapsulated the logic in the capture_output class. The code will be easier to understand.



p = subprocess.Popen(["./subprogram"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout_capture = capture_output(p, capture_stream=p.stdout, output_stream=sys.stdout)
stderr_capture = capture_output(p, capture_stream=p.stderr, output_stream=sys.stderr)
p.wait()

stdout = stdout_capture.output()
stderr = stderr_capture.output()
print("get back control in main flow")

The capture_output class starts a thread. It will read an output specified in capture_stream. It will rewrite this output to another output specified with output_stream. The thread stops when the subroutine ends.



class capture_output:

    def __init__(self, process, capture_stream: Optional[IO] = None, output_stream: Optional[IO] = None):
        self.capture_logs = []
        self.subprocess = process
        self.capture_stream = capture_stream
        self.output_stream = output_stream
        self.thread = Thread(target=self._run_capture)
        self.thread.start()

    def _run_capture(self):
        if self.capture_stream is not None:
            while True:
                line = self.capture_stream.readline().decode('utf-8')

                if line != '':
                    self.output_stream.write(line)
                    self.output_stream.flush()
                    self.capture_logs.append(line)

                if self.subprocess.poll() is not None:
                    break

    def output(self):
        self.thread.join()
        return '\n'.join(self.capture_logs)

Here is the demo of the script.

This method works on Linux. It's time to test it on Windows.

modify the subroutine for windows

We will use a batch script instead of a shell script. I want to run this code in a CMD terminal. I could have used a shell port like git bash. It seemed like cheating to me.

./subprogram.bat



set /p name=What is your name ?

echo Hello %name%
sleep 3
echo Hello World !

Instead of calling .subprogram.sh, we call subprogram.bat. The rest of the code is identical.



p = subprocess.Popen(["subprogram.bat"], stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=os.getcwd())
stdout_capture = capture_output(p, p.stdout, sys.stdout)
stderr_capture = capture_output(p, p.stderr, sys.stderr)
p.wait()

stdout = stdout_capture.output()
stderr = stderr_capture.output()
print("get back control in main flow")

Here is the demo. It's exactly the same as on Linux.

To conclude

You've reached the end. If you have any remaining questions, come share them in the comments. I will enjoy reading and responding to them.

This solution opens up new possibilities for alfred. Capturing output is easier to process in build scripts. It will be possible to run a python server and a nodejs server in a single command. It will show both outputs in the same terminal.

I still need to improve this solution. Interactive programs, those that have color-rich prompts, run in gradients through Popen. I want to address this point but the challenge is on another level. I have the impression that it requires taking an interest in pseudo terminals (pty).

Are you looking for an alternative to make and bash scripts to automate actions in your project ? Come try alfred. The tool might surprise you.command.

References

Inspect a dynamic Ansible AWS inventory

Fabien Arcellier — Fri, 18 Aug 2023 13:00:00 +0000

We will discover the power of ansible-inventory to inspect the structure of the inventory built by ansible when the inventory is generated from the metadata of a machine park. No more wasted time figuring out which groups and hosts Ansible referenced when loading a dynamic inventory.

Ansible is able to generate dynamic inventory from an AWS account. The aws_ec2 plugin builds the inventory from the metadata it finds on the account.

To use it, you must configure the aws_ec2 plugin in an inventory file.

ansible -i aws_ec2.yml --list-hosts all

  hosts (2):
    ec2-52-XX-XX-25.eu-west-1.compute.amazonaws.com
    ec2-52-XX-XX-48.eu-west-1.compute.amazonaws.com

[WARNING]: Could not match supplied host pattern

It all starts with the error WARNING]: Could not match supplied host pattern. This error message is generated when ansible cannot find a match between groups and hosts referenced in the inventory.

ansible-playbook -i aws_ec2.yml deploy.yml

[WARNING]: Could not match supplied host pattern, ignoring: farcellier.com

PLAY [farcellier.com] *******************************************************************************************
skipping: no hosts matched

I wasted an hour trying to figure out why ansible couldn't find the farcellier.com group in the inventory when it should have been there. I would have saved time if I knew how to inspect the inventory.

Inspect inventory

The ansible-inventory command allows you to inspect the inventory as ansible sees it. It's a release to be able to see what ansible is interpreting instead of trying to guess.

ansible-inventory -i aws_ec2.yml --graph

@all:
  |--@aws_ec2:
  |  |--ec2-52-XX-XX-25.eu-west-1.compute.amazonaws.com
  |  |--ec2-52-XX-XX-48.eu-west-1.compute.amazonaws.com
  |--@other_website:
  |  |--ec2-52-XX-XX-25.eu-west-1.compute.amazonaws.com
  |--@farcellier_com:
  |  |--ec2-52-XX-XX-48.eu-west-1.compute.amazonaws.com
  |--@ungrouped:

The solution is before our eyes. Ansible renames the farcellier.com tag to farcellier_com.

Inspect inventory deeply

The ansible-inventory command allows you to see even more. It is able to trace all the variables captured at the host level. This is handy for improving the AWS inventory definition file.

ansible-inventory -i aws_ec2.yml --list

I've posted a snippet of all the variables that are fetched.

{
  "_meta": {
    "hostvars": {
      "ec2-52-XX-XX-25.eu-west-1.compute.amazonaws.com": {
        "ami_launch_index": 0,
        "ansible_host": "ec2-52-XX-XX-25.eu-west-1.compute.amazonaws.com",
        "architecture": "x86_64"
      },
      "ec2-52-XX-XX-48.eu-west-1.compute.amazonaws.com": {
        "ami_launch_index": 0,
        "ansible_host": "ec2-52-XX-XX-48.eu-west-1.compute.amazonaws.com",
        "architecture": "x86_64"
      }
    }
  },
  "all": {
    "children": [
      "aws_ec2",
      "other_website",
      "farcellier_com",
      "ungrouped"
    ]
  },
  "aws_ec2": {
    "hosts": [
      "ec2-52-XX-XX-25.eu-west-1.compute.amazonaws.com",
      "ec2-52-XX-XX-48.eu-west-1.compute.amazonaws.com"
    ]
  },
  "other_website": {
    "hosts": [
      "ec2-52-XX-XX-48.eu-west-1.compute.amazonaws.com"
    ]
  },
  "farcellier_com": {
    "hosts": [
      "ec2-52-XX-XX-25.eu-west-1.compute.amazonaws.com"
    ]
  }
}

References

Ansible AWS inventory used in this blog post

For reference, I share the inventory I used. It selects machines in the eu-west-1 region, groups them from the Name tag.

aws_ec2.yml

---
plugin: aws_ec2

regions:
  - eu-west-1

filters:
  # All instances with their state as `running`
  instance-state-name: running

keyed_groups:
 - key: tags.Name
   separator: ''
   prefix: ''

compose:
    ansible_host: public_dns_name

build custom color template in draw.io

Fabien Arcellier — Tue, 08 Aug 2023 11:35:00 +0000

draw.io is one of my favorite tools for iterating and talking about software and system architecture diagrams. It's intuitive, allows real-time drawing, provides a library of ready-to-use icons, and even works offline.

However, there's one drawback when it comes to using these diagrams in corporate documents: they often end up looking unprofessional because of draw.io's default style. Finding the perfect colors becomes a tedious process, there's no way to save them, and the diagrams end up feeling generic.

This problem had been bugging me for years. But guess what? I recently discovered it could be tune with just a few lines of json. Let's see how.

draw.io interface with custom colors, default style, ...

the default palette is so comfortable

With just a few clicks, draw.io allows you to create high-quality diagrams. You simply choose shapes from the library, drag them onto the canvas, and connect them together. Even better, you can instantly customize them using the default palette.

However, there's a downside: all the diagrams start to look the same after a while. To stand out from the crowd, you need to create your own libraries and spend a considerable amount of time customizing colors. It can be tedious to start from scratch for each diagram and try to find the right colors...

Enrich the palette with your own styles

Il est temps de rentrer dans la configuration de draw.io. Elle s'écrit en json.

Elle va nous permettre d'ajouter nos propres couleurs. Nous conserverons le même confort d'utilisation avec elles qu'avec celles de la palette par défaut.

By pasting the following code into Extras > Configurations and restarting draw.io, our new palette will be there.



{
  "customColorSchemes": [
    [
      {
        "fill": "#ffffff",
        "stroke": "#EA7317",
        "font": "#EA7317"
      },
      {
        "fill": "#ffffff",
        "stroke": "#1061a4",
        "font": "#1061a4"
      },
      {
        "fill": "#ffffff",
        "stroke": "#178bea",
        "font": "#178bea"
      },
      {
        "fill": "#ffffff",
        "stroke": "#eac117",
        "font": "#eac117"
      },
      {
        "fill": "#ffffff",
        "stroke": "#a48710",
        "font": "#a48710"
      },
      {
        "fill": "#EA7317",
        "stroke": "none",
        "font": "#ffffff"
      },
      {
        "fill": "#1061a4",
        "stroke": "#ffffff",
        "font": "#ffffff"
      },
      {
        "fill": "#178bea",
        "stroke": "none",
        "font": "#ffffff"
      },
      {
        "fill": "#ffffff",
        "stroke": "none"
      }
    ]
  ]
}

Here are notable elements of this configuration.

customColorSchemes allows to describe several palettes. The palettes are displayed one after the other.
a color palette can contain many colors, here is an example with 14 color sets

each entity accepts a fill attribute which defines the fill color of the shapes
each entity accepts a stroke attribute which defines the stroke color of the shapes
each entity accepts a font attribute which defines the font color
the color will be transparent if an attribute is set to "none"

Here is the rendering of the styles we just loaded.

Enrich Color Picker

The color picker can also be enriched. Our colors will be on hand whenever we need them.

The configuration is done in the same place as for the color palettes.



{
  "customColorSchemes": [
    [
      {
        "fill": "#ffffff",
        "stroke": "#EA7317",
        "font": "#EA7317"
      },
      {
        "fill": "#ffffff",
        "stroke": "#1061a4",
        "font": "#1061a4"
      }
    ]
  ],
  "presetColors": [
    "ffffff",
    "EA7317",
    "1061a4",
    "178bea",
    "eac117",
    "a48710",
    "EA7317",
    "a45b10",
    "eaac17",
    "ea4417",
    "a42f10"
  ]
}

presetColors is a color array expressed in hexadecimal without the `` prefix. If the prefix is present, the color is ignored. ## Configurer un style par défaut sur tout le diagramme

Integrate a design system in draw.io

Configuring the color palette is a first step. It is possible to go further. Other attributes such as font, style of arrows, ... can be configured this way. There is more information on the official documentation.

It's such a comfort to have everything at your fingertips. There are other facets of draw.io that can be customized.

Here is a non-exhaustive list that can be customized:

import / export an icon library
import / export your own shapes
import / export your own parametric shapes
use a specific font
use a default style
declare your own diagram styles
...

Are there any that you would like me to detail in a future article? Are there any other assets you customize in Draw.io? Come and share your experiences in the comments.

References

package a poetry project in a docker container for production

Fabien Arcellier — Mon, 30 Jan 2023 13:10:00 +0000

Docker become the preferred mode of application distribution. For python applications it has become mandatory because it virtually guarantees that if an image works on one environment, then it will work on the others. This strategy also makes it possible to finely manage deployments. Docker allows the promotion of an image from environment to environment by playing on version number and allow devops practices as gitops or canary deployment.

As I explain in the article I move from pipenv to poetry in 2023 - Am I right ?, I move my projects from pipenv to poetry. I also had to upgrade the docker packaging. I will try to share what I learn. I will describe 5 practices I've used to build docker images from poetry projects.

install a virtual environment in the docker image
copy only what is strictly necessary by filtering files and folders with .dockerignore
ignore dev libraries when installing dependencies
add the virtual environment in the PATH
use a multi-stage docker build to exclude poetry

install a virtual environment in the docker image

In docker image, we will install a virtual environment in the project as we would locally. The environment will be installed in /app/.venv folder. Doing that is required to be able to exclude poetry dependency thank to a docker multi-stage build.

To ensure that the virtual environment of a project is created next to the sources, in a predictable folder, poetry must be configured. We can do it with this command:

poetry config virtualenvs.in-project true --local

I have doubt first on this practice. I had the impression that creating a virtual environment in a container is counterproductive. However, it was necessary to be able to exclude poetry from the final image.

copy only what is strictly necessary by filtering files and folders with .dockerignore

In a production image, it is not a good idea to drag all files and folders that are used for development. The .dockerignore file is a manifest that specifies files and folders to exclude when adding a folder to an image with the ADD or COPY statement. The .dockerignore uses the same syntax as gitignore.

In general, we'll ignore the package that contains the tests, build artifacts, .venv folder. You can ignore all artifacts that are produced by utilities like mypy, coverage, ...

.dockerignore

/.venv
/tests
/build

Here is an example dockerfile. The code is deployed in the /app folder. The COPY . /app does not copy all items specified in manifest .dockerignore

Dockerfile

FROM python:3.10-slim

RUN pip install poetry  
RUN mkdir -p /app  
COPY . /app

WORKDIR /app

RUN poetry install
CMD ["poetry", "run", "python", "-m", "app.main"]

ignore dev libraries when installing dependencies

poetry allows us to install project dependencies while ignoring development dependencies. These dependencies take up space and are unnecessary when the application is running in production. In addition, they remain accessible at runtime and can degrade the security of your application.

FROM python:3.10-slim

RUN pip install poetry  
RUN mkdir -p /app  
COPY . /app

WORKDIR /app

RUN poetry install --without dev
CMD ["poetry", "run", "python", "-m", "app.main"]

add the virtual environment in the PATH

By default, the docker image uses system python. It's not practical to always prefix the command with poetry run. We will add the python of the virtual environment of poetry to the PATH to use it in priority.

This trick makes the container more robust because you won't need to think about poetry anymore to place a custom command to the container.

# ...
ENV PATH="/app/.venv/bin:$PATH"

RUN poetry install --without dev
CMD ["python", "-m", "app.main"]

I prefer this technique to the entrypoint overload because it makes using commands like sh or bash intuitive when we need to connect on the container.

use a multi-stage docker build to exclude poetry

This practice makes it possible to exclude all the libraries used during the build. The container environment will no longer contain poetry or the dependencies that this toolkit pulls.

This practice has the same effect as avoiding installing dev dependencies. It makes it possible to reduce the size of the container, to pass it from a container of 240Mb to 144Mb.

FROM python:3.10-slim as builder

RUN pip install poetry
RUN mkdir -p /app
COPY . /app

WORKDIR /app
RUN poetry install --without dev

FROM python:3.10-slim as base

COPY --from=builder /app /app

WORKDIR /app
ENV PATH="/app/.venv/bin:$PATH"
CMD ["python", "-m", "app.main"]

These 5 practices may help you build your workflow to distribute a docker image from a poetry project. If you use other workflows or tips on a daily basis, share them with us in the comments.

Finally, I have discarded several practices from this article that you can also apply.

You can make your own base image instead of using an already packaged base image. The python image is updated regularly. It has positive effects like being up to date but also negative like redownloading all the layers at each deployment. In a context where bandwidth is precious, it is interesting to better plan the update of the base image to reduce bandwidth consumption.

In the same vein, another practice is to install external dependencies and application code separately. It's always the same desire to save bandwidth by only downloading the layer that has been modified.

A last practice that I do not use at all and which may interest you is to use slim toolkit to keep only the useful elements in your final image.

I move from pipenv to poetry in 2023 - Am I right ?

Fabien Arcellier — Tue, 03 Jan 2023 14:36:00 +0000

Package management in python is evolving continuously. This evolution is strewn with initiatives of all kinds, each with their pitfalls. It's frustrating to see all these ways of doing things coexist, to have to question what we know about the subject every year. At the same time, it is the mark of an active community that is looking for new ways of doing things, an environment that evolves with the times. It reminds us of this long transition that took place over more than 10 years to go from python 2 to python 3. It was a hard choice but come with great benefits.

In 2019, I chose to use the pipenv + setuptools combination as a package manager. This combination has had served its time. I decided at the end of 2022 to migrate all the projects I work on to poetry.

I still have doubts while writing this article. The purpose of this article is to share my knowledge and my findings at the time of writing. I can be wrong or even make false assumptions. Share in comments if you have a different vision, and share in comments in you know interesting PEP (python enhancement proposal) on this topic not mentioned in this article.

What am I expecting from a package manager at the end of 2022?
- standard manifest
- manifest scaffholding
- virtual environment (.venv layout)
- dependencies locking
- update of one / all dependencies
- organization of the source code with src layout
- sharing a library on pypi
- version management
- multiple python projects in a mono repository
Moving to poetry

What am I expecting from a package manager at the end of 2022?

A python project without a manifest is a car without air conditioning. After having tasted it, it is difficult to do without it at certain key moments.

A manifest contains the declaration of dependencies, the version number, the exposed python packages, the development dependencies and a whole bunch of other metadata... The manifest is mandatory to publish a library on pypi. It avoids a lot of headaches when the project grows, for example the hell of relative packages or dependencies that evolve and become incompatible with each other.



from .util import filter
from ..core import local

The only problem is that each package manager has its own manifest format. Here are the manifests that you may have come across and which are more or less exclusive: the requirement.txt, pyproject.toml, setup.cfg, setup.py and Pipfile

A package manager may transpile its manifest to other formats as needed. For example, I think that poetry transpiles the pyproject.toml to setup.py manifest to build a wheel, the format expected by pypi to distribute a library, and publish it. It's transparent to us.

In 2022, a package manager is a tool that from a manifest will install dependencies, manage a virtual environment, lock dependencies to a given version, ... Things that at first are optional in python.

Here is a grid analysis that represents my expectations for a package manager in 2023 with a subjective evaluation of the benefits of each combination.

	Poetry	pipenv + setuptools	pipenv	setuptools
standard manifest	4	4	2	5
manifest scaffholding	5	1	3	0
virtual environment (.venv layout)	5	3	3	0
dependencies locking	5	3	3	0
update locked dependencies	5	2	2	0
source code organization (src layout)	4	4	(?)	4
publication of library	4	3	0	4
version management	3	4	3	4
mono-repository	1	1	0	0

projects that use setuptools : requests, django, flask, boto3...
projects that use poetry : ward, ... (if you have some with more than 500 stars, share them in comments)
projects that use pipenv : ... (if you have some with more than 500 stars, share them in comments)

setuptools is awesome, but it is doing only one things, but it is doing it almost perfectly. Unfortunately, it miss features I want to have in my project as managed virtual environment, locked dependencies, ... I won't talk about it in this article.

standard manifest

The pyproject.toml manifest has been official since PEP 621. It is still young. poetry uses this format with its own guidelines. setuptools is supported it for few months. The support is pretty good but there is some limitation. I tried this manifest two months ago and I fail to install a python application in editable mode with pyproject.toml manifest. I don't know if it's a limitation of pip or setuptools, but it's a problem for development.

pipenv uses the Pipfile format. He is the only one to use it. A Pipfile manifest can reference a setup.cfg or setup.py manifest. It is this feature that led me 2 years ago to choose pipenv + setuptools.

manifest scaffholding

I instantiate a manifest with every python project I start. With setuptools, I have to write this manifest by hand. The scaffholding brings together assistants who assist us in writing a manifesto. This is the case with poetry with the init command. pipenv also offers scaffholding but on the Pipfile manifest.



poetry init

The package manager can offer a more advanced scaffholding. poetry can declare new dependencies in the manifest with poetry add.



poetry add requests
poetry add --dev pytest

The scaffholding is essential for a newcomer who comes from nodejs or java. This is his first contact with the environment. This is the guarantee of creating a python project that can be used by other developers without reading 10 tutorials that contradict each other...

pyproject.toml manifest



[tool.poetry]
name = "myproject"
version = "1.0.0"
description = "this awesome project"
authors = ["Fabien Arcellier <>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.8"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

virtual environment (.venv layout)

pipenv and poetry share the same DNA. The 2 tools bind a virtual environment to a python project. A virtual environment, independent of the python installed on the system, is automatically associated with the manifest and mounted as soon as the user issues a command in the package manager, for example poetry install.

On all the projects that I have initiated for 2 years, I use the .venv convention. the virtual environment is created at the root of the python project in .venv directory. poetry uses the default .venv layout. As soon as you create a project with poetry init, the associated virtual environment is automatically declared in the .venv folder next to pyproject.toml. This convention allows me to share run configurations in the IDE with other developers, whether on pycharm or vscode.



.
├── pyproject.toml
└── .venv
    ├── bin
    ├── .gitignore
    ├── lib
    └── pyvenv.cfg

dependencies locking

locking dependencies ensures that all dependencies will be reinstalled with the same version regardless of which developer clones the project locally.

I happened to clone a repo several months after the last commit and noticed dependency problems because a 2nd or 3rd level library evolved and this evolution "broke" a level 1 library that uses it. It's a frustrating problem. I had the case this year with label-studio-converter one of the repo of the excellent label-studio

Dependency locking is provided by poetry and pipenv. poetry is faster and seems more robust.
pipenv fails to install dependencies on machine learning pocs that used pytorch or yolo.

poetry.lock



[[package]]
name = "altair"
version = "4.2.0"
description = "Altair: A declarative statistical visualization library for Python."
category = "main"
optional = false
python-versions = ">=3.7"

[package.dependencies]
entrypoints = "*"
jinja2 = "*"
jsonschema = ">=3.0"
numpy = "*"
pandas = ">=0.18"
toolz = "*"

Locking dependencies is a plus to guarantee the deployment on a PaaS hosting that deploys source code instead of docker image like heroku, .... It guarantees the installation in production of the same dependencies that we use locally in our developments.

update of one / all dependencies

Once the dependencies are locked, we are guaranteed to always install the same versions, even a year later. However, it may become required to update these dependencies. poetry does this with poetry update command. Note that this command only updates dependencies locked in the poetry.lock file and does not touch the pyproject.toml manifest.

poetry also allows you to update just one library rather than all libraries in the project at once, for example, just requests. It allows more flexibility in the update policy, especially on security alerts that can be raised by github for example. This is something I missed with pipenv+setuptools. I had to update all the locked dependencies at once.

organization of the source code with src layout

Relative imports are bad practice in python. This pattern feel wrong, you can import a module of one directory above but not more.



# to avoid
from .util import foo

# to do
from myproject.util import foo

The project manifest definitely solves this problem. The package manager installs our project in the virtual environment and allows us to use our modules like the other modules of the standard library from their full path.

For 2 years, I have the impression that the src layout takes precedence over other methods of organizing source code such as the flat layout. (see discussion src layout vs flat layout).

A project organized according to src layout implements the packages in a src folder. The layout looks like the following.



.
├── poetry.lock
├── pyproject.toml
├── README.md
├── src
│   ├── my_project
│   │   ├── __init.py__
│   │   └── util.py 
└── tests

Package are declared in the manifest pyproject.toml with the directive packages.

pyproject.toml



[tool.poetry]
name = "myproject"
version = "1.0.0"
description = "this awesome project"
authors = ["Fabien Arcellier <>"]
readme = "README.md"
packages = [
  { include = "my_project", from = "src" }
]

[tool.poetry.dependencies]
python = "^3.8"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

src layout seems to be supported by all package managers, including poetry. This layout is not linked to a PEP. I don't know yet if this trend will continue. It is similar to what is found in nodejs or java. I find that it facilitates the organization of projects in python, makes it easier to get started with the project. Today, I apply it in libraries, web projects like flask or streamlit but it's one of the trends that could fall. I kept the flat layout only with django. manager.py does not work easily with src layout.

sharing a library on pypi

I think that having the ability to share a native python library helps to democratize python and encourage sharing in the community. Everyone can share their work. The overall quality is not terrible, but this ocean of contribution pulls the ecosystem upwards and encourages its evolution.

poetry allows you to publish a library on pypi with the poetry publish command. For pipenv + setuptools , I had to install twine in the development dependencies and use it to orchestrate the release to pypi. Even if the quality is there, I had to find out how to organize my manifesto, how to use twine, how to industrialize it before being able to share my projects.

version management

The version management in poetry is poorer than with setuptools. You must write the version number in the manifest, not elsewhere. setuptools has more options to retrieve this version number, either from a file or from an attribute in a module. This evolution makes all the same sense because the manifest is the ideal place to record a version number.

I use the version number of a python package to automatically create a tag in git and trigger a pipeline in the CI/CD. Here is the snippet, compatible with poetry, that I use to retrieve the version number of a package and then create a corresponding git tag.



import fixtup

ROOT_DIR = os.path.realpath(os.path.join(file, "..", ".."))

VERSION = importlib.metadata.version(fixtup.name)

multiple python projects in a mono repository

In enterprise repos, I use a mono-repository architecture with multiple python projects in a single repo. Above, I only have partial answers. poetry does not offer a workflow for managing multiple python projects. I use either a homemade utility alfred-cli to manage the mono-repository, bash scripts, or simply a makefile. No solution fully meets my requirements.

At the moment, alfred-cli is an open source experimental project I have written but its documentation and design is too messy.

Moving to poetry

In 2022, with the standardization of the pyproject.toml manifest, poetry stands out as the most robust and easy-to-use package manager, even for a newcomer to the python universe. In all aspects, it is better than pipenv with a developer experience that I find more complete and enjoyable.

	Poetry	pipenv + setuptools	pipenv	setuptools
standard manifest	4	4	2	5
manifest scaffholding	5	1	3	0
virtual environment (.venv layout)	5	3	3	0
dependencies locking	5	3	3	0
update locked dependencies	5	2	2	0
source code organization (src layout)	4	4	(?)	4
publication of library	4	3	0	4
version management	3	4	3	4
mono-repository	1	1	0	0

poetry comes from the experience accumulated by decades of wandering around package management in python. I regret leaving setuptools. It is a robust and powerful utility which in its time also made a huge difference. Its development is active and it masterfully ensures compatibility between several generations of manifest.

I think this is the right direction to take for the next 1-2 years given the current ecosystem. I may have blind spots on this. Did you make the transition yourself ? Did you choose another package manager for your projects ? Share your experience in comments

cover image by Eli Digital Creative from Pixabay

Test beyond your code with docker & pytest

Fabien Arcellier — Sat, 17 Dec 2022 10:50:06 +0000

Docker opens up access to an incredible ecosystem of build-up applications. We find containers for everything … We can run a postgres database, run a redis database or launch an ftp server.

Docker is a great way to run dependency our code needs when running system integration tests that will validate our application.

In this article, we will discover how to use docker in our python tests by writing a minimum of boilerplate while keeping the same CI/CD pipeline.

Testing an application beyond its code thanks to docker

A bug can hide in the use of a database, for example, a badly written SQL query. We need a database to test that. We can’t validate this behavior with just the code.

It is an important topic. Django, a major python framework, provides a partial answer to this. It only manages a test workflow for the database. This workflow also requires you to launch the database by yourself before launching your tests.

To solve this topic, our tests must be able to launch containers by themselves.

To mount the containers from our tests, we will boost our tests with Fixtup. Our tests will be able to start and stop containers at the right time by themselves.

If a container is expensive to start, it can be mounted once and reused. pytest will stop it after playing the last test it has to play.

Install the requirements

To follow this article, we need to prepare our developper station.

1 . We need to be able to use docker and docker-compose from a terminal.

$ docker --version
$ docker-compose --version

2 . We need poetry to create the python project

$ poetry

Create a python project

$ poetry init
$ poetry add psycopg2-binary
$ poetry add --dev pytest fixtup

We will create a docker container for postgresql as a fixture to use in our tests.

$ poetry run fixtup init
Choose a directory to store fixture templates : tests/fixtures
Python manifest (pyproject.toml) [pyproject.toml]

$ poetry run fixtup new
Choose a fixture identifier : postgres
Mount environment variables on this fixture (y/n) [n]
Mount docker container on this fixture (y/n) [n] y

tests/fixtures/postgres/docker-compose.yml

version: "3.9"
services:
    postgresql:
        init: true
        image: postgres
        ports:
            - "5432:5432"
        environment:
            - POSTGRES_PASSWORD=1234
        volumes:
            - /var/lib/postgresql

We will activate the hook_started hook to wait for the database server to be loaded before starting our test.

tests/fixtures/postgres/.hooks/hook_started.py

from time import sleep
import fixtup. helper

fixtup.helper.wait_port(5432, timeout=2000)

# we wait 1 second to ensure the database is ready
sleep(1)

Using postgresql database in a test

Now we are going to install the driver for postgresql and write our first test.

test/test_postgresql_database.py

import fixtup
import psycopg2

def test_postgres_should_work():
    with fixtup.up('postgres'):
        conn = psycopg2.connect("host=127.0.0.1 dbname=postgres user=postgres password=1234")
        core = conn.cursor()
        cur.execute('SELECT 1+1')
        res = cur.fetchall()
        assert res[0][0] == 2
        conn.close()

Use the same postgresql database for all tests

Containers can take a long time to start, that’s why we would like to reuse them between several tests. By default, fixtup stops and starts them between each test. By activating the keep_up policy of fixtup, the containers will remain up as soon as they have been launched. They will be deleted when pytest has finished running.

tests/fixtures/postgres/fixtup.yml

# This flag control if a fixture stay up and running between every test. The fixture is stop and
# unmount when the test process stop
#
# This attribute allow to start a database only once and stop the container only when unittest has finished to run
# the test suite. It may be interested to improve the performance if your start and stop process is too slow
keep_up: true

Wait for database

Instead of waiting for port availability and a timer after to wait database to be ready, we will rewrite the hook_started to attempt to connect to the database regularly.

tests/fixtures/postgres/.hooks/hooks_started.py

from time import monotonic
import psycopg2

start = monotonic()
connected = False
timeout = 5
while not connected:
    try:
        conn = psycopg2.connect("host=127.0.0.1 dbname=postgres user=postgres password=1234")
        connected = True
        conn.close()
    except Exception:
        if monotonic() - start > timeout:
            raise TimeoutError()

Going further

In this article, we have seen how to use docker containers in our tests with pytest and fixtup.

If your test depends on an application that works in docker, pytest can mount these containers through Fixtup. It can be a redis base, a localstack stack for AWS, or an ftp server...

If you're stepping through a test from pycharm or vscode, you can debug your test and also the fixture hooks.

We haven't seen how to mount the database schema at startup, nor how to clean up the database between each test. Fixtup hooks allow you to do this.

I plan to develop a fixtup plugin for sqlalchymie and another for django that will allow you to use a database without writing a specific hook.

wait for a server to respond in python

Fabien Arcellier — Sat, 10 Dec 2022 05:00:00 +0000

When starting a python application, our application may wait for a server to respond.

This article introduce a common technique to wait for a server to be available using socket from python stdlib. A script may wait for a database to start or for an API to become available.

I have written wait_port, the following function, for the open source library fixtup. fixtup starts containers so that a test can run. There is a need to wait before returning control or executing migration code until the container is operational. The wait_port helper allows you to wait in a hook before returning control to pytest.

# wait postgresql to start until 5 seconds or raise Timeout exception
wait_port(5432, timeout=5000)

here is the snippet you can reuse in your project

def wait_port(port: int, host: str = 'localhost', timeout: Optional[int] = None, attempt_every: int = 100) -> None:
    """
    wait until a port would be open, for example the port 5432 for postgresql
    before going further

    >>> fixtup.helper.wait_port(5432, timeout=5000)

    :param port: port that has to be open
    :param remote_ip: host on which the port has to be open. It will be localhost by default
    :param timeout: timeout in ms before raising TimeoutError.
    :param attempt_every: time in ms between each attempt to check if the port is responding
    """
    start = monotonic()
    connected = False
    while not connected:
        with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
            try:
                s.connect((host, port))
                connected = True
            except ConnectionRefusedError:
                if timeout is not None and monotonic() - start > (timeout / 1000):
                    raise TimeoutError()

        sleep(attempt_every / 1000)

We can go further with the principle of this snippet. It can be used to verify that a database is ready for use by checking for the existence of a table or by executing a SELECT 1 + 1 query instead of connecting with a socket.

If you had to write resilient code to wait for a server, share your experience with us in the comments.

test a ftp data pipeline with pytest & docker

Fabien Arcellier — Mon, 05 Dec 2022 14:55:00 +0000

System integration tests are difficult to write, difficult to run in a continuous integration platform.

That’s behind us. I will convince you that they are easier to write and run in a continuous integration platform than you might think. We are going to write together the system integration tests of a data pipeline that downloads data from open weather and deposits it on an ftp server.

Fixtup is an open source python library for testing a python application beyond code boundaries. It manages the creation of disposable environments and is compatible with all modern CI platforms that support docker-compose such as gitlab ci, github action, etc… The library works with pytest and unittest and runs on linux, macOS and windows.

At the end of this article, you will know how to test a data pipeline in python with pytest and fixtup, how to write system integration tests which verify the behavior of your code on the external resources and will reduce the apprehension of the next developer who will have to upgrade your stack …

Ftp server still here in 2022

The integration of data to an ftp server is omnipresent in the industry, even in 2022. An FTP server is an easy to operate, robust and easy to deploy solution for sharing data. Everyone knows how to connect, read and write to an FTP server.

No standard has managed to relegate it to the closet. The FTP protocol works rather in synergy with its replacements, for example cloud storage solutions like AWS S3, Azure Blob storage or Google Cloud Storage.

In this article, the data pipeline will be an ETL that regularly publishes a dataset to an ftp server.

Write a data pipeline

This data pipeline retrieves wind and humidity every minute from the Open Weather API for the city of Lyon in France and publishes them every 10 minutes on an ftp in the form of a csv file.

	ok	wind_speed	humidity
2022-03-11T12:01	1	15	0.1
2022-03-11T12:02	1	13	4
…
2022-03-11T12:08	0
2022-03-11T12:09	1	4	4.1
2022-03-11T12:09	1	4	4.1

It exports a csv file every 10 minutes. It writes the file with a name in the form file-2022-03-11T12-10.csv for the increment between 03/11/2022 between 12:01 and 12:10. A file does not change once it has been written.

Test the data producer with a real ftp server

System integration tests validate the behavior of a module with an external system and can check the non-regression of the library over time.

These tests offer guarantees when the version of python evolves, or when the ftputil library needs to be updated. It is safe to say that simple write operations on an external system are validated on a real ftp server by the continuous integration platform.

Test the data producer with pytest and fixtup

Fixtup uses docker to mount disposable environments. It build a complete environment in one line in your test and can debug one or more tests in pycharm or vscode without worrying about having to start the environment.

def test_that_mount_a_ftp_server():
    with fixtup.up('ftp_server'):
        # Assign
        # Acts
        # Assert
        pass

Write a dataset on the FTP server

Our ETL code downloads data from openweather every minute and writes it every 10 minutes to an ftp server.

Here is a code extract from the ETL. You will find the complete code in the example of fixtup repo.

@dataclasses.dataclass
class OpenWeatherDataset:
    records: List[Tuple[datetime, Optional[dict]]] = dataclasses.field(default_factory=list)

    def enqueue(self, time: datetime, record: dict):
        self.records.append((time, record))

    def pop(self) -> List[Tuple[datetime, dict]]:
        records = self.records
        self.records = []

        return records

def pipeline_weather_publication(dataset: OpenWeatherDataset):
    records = dataset.pop()

    ftp_host = os.getenv('FTP_HOST', 'localhost')
    ftp_user = os.getenv('FTP_USER', None)
    ftp_pwd = os.getenv('FTP_PASS', None)

    fieldnames = ['timestamp', 'measure_timestamp', 'ok', 'wind_speed', 'humidity']
    content = []
    for timestamp, record in records:
        row = {}
        row['timestamp'] = timestamp.isoformat()
        if record is not None:
            row['ok'] = 1
            wind_section = record.get('wind', {})
            row['wind_speed'] = wind_section.get('speed', None)
            row['measure_timestamp'] = record.get('dt', 0)
            main_section = record.get('main', {})
            row['humidity'] = main_section.get("humidity")
        else:
            row['ok'] = 0

        content.append(row)

    now = datetime.utcnow()
    file_timestamp = now - timedelta(minutes=now.minute % 10, microseconds=now.microsecond)
    filename = f"file-{file_timestamp.strftime('%Y-%m-%dT%H-%M')}.csv"

    with tempfile.TemporaryFile('a+') as filep:
        csv_file = csv.DictWriter(filep, fieldnames)
        csv_file.writeheader()
        for row in content:
            csv_file.writerow(row)

        filep.seek(0)

        with ftputil.FTPHost(host=ftp_host, user=ftp_user, passwd=ftp_pwd) as host:
            with host.open(filename, 'w') as ftp_filep:
                ftp_filep.write(filep.read())

test writing a dataset on the ftp

The integration tests will allow us to test that the function responsible for publishing the data on an ftp server writes a dataset in csv format with the expected name format.

def test_pipeline_weather_publication_should_export_a_dataset_on_a_ftp():
    with fixtup.up('ftp_server'):
        # Assign
        dataset = OpenWeatherDataset()
        dataset.enqueue(datetime(2022, 11, 20, 12, 5), {'wind': {'speed': 15}, 'main': {'humidity': 3}})
        dataset.enqueue(datetime(2022, 11, 20, 12, 6), {'wind': {'speed': 15}, 'main': {'humidity': 3}})
        dataset.enqueue(datetime(2022, 11, 20, 12, 7), {'wind': {'speed': 15}, 'main': {'humidity': 3}})
        dataset.enqueue(datetime(2022, 11, 20, 12, 8), {'wind': {'speed': 15}, 'main': {'humidity': 3}})
        dataset.enqueue(datetime(2022, 11, 20, 12, 9), {'wind': {'speed': 15}, 'main': {'humidity': 3}})
        dataset.enqueue(datetime(2022, 11, 20, 12, 10), {'wind': {'speed': 15}, 'main': {'humidity': 3}})

        # Acts
        pipeline_weather_publication(dataset)

        # Assert
        with ftputil.FTPHost(host=os.getenv('FTP_HOST', 'localhost'), user=os.getenv('FTP_USER', None), passwd=os.getenv('FTP_PASS', None)) as host:
            names = host.listdir(host.curdir)
            assert len(names) == 1
            assert 'file-2022-11-20T12-10.csv' in names

When the test runtime enters the fixtup.up instruction, it mounts the docker-compose stack associated with the ftp_server fixture. Fixtup will keep this container up and stop it when all tests have finished running.

Reset ftp content between each test

Between each test, it is better to start with an environment in an identical state. We need to clean the ftp server between each test. Fixtup allows us to write hooks in python for that. The hook_setup_data.py hook runs at the start of each test.

tests/fixtures/ftp_server/.hooks/hook_setup_data.py

import os
from typing import Optional

import ftputil
from ftputil import FTPHost


def ensure_ftp_is_empty(ftp: FTPHost, path: Optional[str]):
    list_doc = ftp.listdir(path)

    for name in list_doc:
        if name != "." and name != "..":
            if ftp.path.isfile(name):
                ftp.remove(name)
            else:
                ftp.rmtree(name)


with ftputil.FTPHost(host=os.getenv('FTP_HOST', 'localhost'), user=os.getenv('FTP_USER', None), passwd=os.getenv('FTP_PASS', None)) as ftp:
    ensure_ftp_is_empty(ftp, ftp.curdir)

In a hook we can also arrange to restore files/folders at the start of each test to start with an already full ftp space.

Fixtup is a complete toolkit to write system integration tests

Upgrading a system without system integration testing is like trying to get a refund for a product without a receipt. It is frustrating and difficult.

Integration testing is a complement to unit testing. They do not replace them. Unit tests validate the behavior of a system up to 100 times faster but are confined to the code level. Fixtup allows you to write tests that can go beyond this boundary.

Fixtup offers tools to control docker containers on which several tests depend and make their execution easier, whether in debug on the developer workstation or in a continuous integration platform such as gitlab ci, github action, etc.

Are you using containers in your CI platform to run system integration tests ? On which technical stack do you think you would need to write system integration tests ? Let us know in the comments below!

To know more

Declare a proxy property to a child object

Fabien Arcellier — Sun, 15 May 2022 15:13:10 +0000

We want to expose an attribute on a facade object, while it is implemented by a child object. Python allows us to do this by declaring a read/write property.

class Foo:

    def __init__(self):
        self.var1 = "hello"

class Bar:

    def __init__(self):
        self.foo = Foo()

    @property
    def var1(self):
        return self.foo.var1

    @var1.setter
    def var1(self, value):
        self.foo.var1 = value

In this code, the var1 property in Bar is a proxy to the var1 attribute of the foo instance.

With this implementation, each time we add an attribute in Foo and want to expose it in Bar, we have to declare 6 lines of codes. This boilerplate will grow and make the class more difficult to read and maintain.

We will implement a descriptor proxy_property to reduce this boilerplate to a single line. Here is the reimplemented Bar class.

class Bar:
    var1 = property_proxy("foo", "var1")

    def __init__(self):
        self.foo = Foo()

A descriptor in python is a class in which we implement the dunder methods __get__, __set__ and __delete__.

Descriptors are a powerful, general purpose protocol. They are the mechanism behind properties, methods, static methods, class methods, and super().

Python Doc - Descriptor HowTo Guide

Here is the implementation of the property_proxy descriptor.

class property_proxy(object):
    """
    A descriptor based recipe that makes it possible to write shorthands
    that forward attribute access from one object onto another.

    >>> class B:
    >>>     def __init__(self):
    >>>         self.foo = 12
    >>>         self.bar = 12
    >>>
    >>> class A:
    >>>     foo: int = property_proxy("b", "foo")
    >>>     bar: int = property_proxy("b", "bar")
    >>>
    >>>     def __init__(self):
    >>>         self.b = B()
    >>>
    >>> a = A()
    >>> print(a.foo)

    This descriptor avoids writing the code below to establish a proxy
     with a child instance

    >>> class B:
    >>>     def __init__(self):
    >>>         self.foo = 12
    >>>
    >>> class A:
    >>>
    >>>     def __init__(self):
    >>>         self.b = B()
    >>>
    >>>     @property
    >>>     def foo(self):
    >>>         return self.b.foo
    >>>
    >>>     @foo.setter
    >>>     def foo(self, value):
    >>>         self.b.foo = value
    >>>

    """

    def __init__(self, objectName, attrName):
        self.objectName = objectName
        self.attrName = attrName

    def __get__(self, instance, owner=None):
        proxy = getattr(instance, self.objectName)
        if proxy is None:
            raise ValueError(f"{self.objectName} does not exists on {instance}")

        return getattr(proxy, self.attrName)

    def __set__(self, instance, value):
        proxy = getattr(instance, self.objectName)
        if proxy is None:
            raise ValueError(f"{self.objectName} does not exists on {instance}")

        setattr(proxy, self.attrName, value)

To conclude

In this post, we found that exposing a child object's attribute in a facade would take a large boilerplate from us and interfere with reading and maintaining our facade.

We implemented a property_proxy descriptor to make this declaration in one line and improve code consistency. The gain in readability is significant.

class Bar:
    var1 = property_proxy("foo", "var1")

    def __init__(self):
        self.foo = Foo()

Limit

The property_proxy declaration has 2 flaws which affect the maintainability of the code. This declaration does not allow a linter like mypy to check the existence and typing of attributes. The IDE's refactoring of an attribute name may miss this declaration because it will consider the attributes in the property_proxy declaration as plain text.

Python Doc - Descriptor HowTo Guide

Forem: Fabien Arcellier

capture and stream the standard output of a process with python on linux and windows

Handling standard outputs

write a subroutine

run this subroutine and let the parent shell handle standard output

run a program and capture standard output

readings on standard output and error output block each other

capture both standard output and error output and display them in real time

modify the subroutine for windows

To conclude

References

Inspect a dynamic Ansible AWS inventory

[WARNING]: Could not match supplied host pattern

Inspect inventory

Inspect inventory deeply

References

Ansible AWS inventory used in this blog post

build custom color template in draw.io

the default palette is so comfortable

Enrich the palette with your own styles

Enrich Color Picker

Integrate a design system in draw.io

References

package a poetry project in a docker container for production

install a virtual environment in the docker image

copy only what is strictly necessary by filtering files and folders with .dockerignore

ignore dev libraries when installing dependencies

add the virtual environment in the PATH

use a multi-stage docker build to exclude poetry

I move from pipenv to poetry in 2023 - Am I right ?

What am I expecting from a package manager at the end of 2022?

standard manifest

manifest scaffholding

virtual environment (.venv layout)

dependencies locking

update of one / all dependencies

organization of the source code with src layout

sharing a library on pypi

version management

multiple python projects in a mono repository

Moving to poetry

Test beyond your code with docker & pytest

Testing an application beyond its code thanks to docker

Install the requirements

Create a python project

Using postgresql database in a test

Use the same postgresql database for all tests

Wait for database

Going further

wait for a server to respond in python

test a ftp data pipeline with pytest & docker

Ftp server still here in 2022

Write a data pipeline

Test the data producer with a real ftp server

Test the data producer with pytest and fixtup

Write a dataset on the FTP server

test writing a dataset on the ftp

Reset ftp content between each test

Fixtup is a complete toolkit to write system integration tests

To know more

Declare a proxy property to a child object

To conclude

Limit

Read more