Forem: Iacopo Spalletti

Cleaning up you test suite with pytest parametrize

Iacopo Spalletti — Sat, 26 Dec 2020 07:48:49 +0000

I admit I jumped on the pytest ship quite late, after a few years of relying on the good ol’ unittest module, and for a good part they worked reasonably well for me, for little apparent incentive to something else.

Over time I started using pytest more and more as I discovered quite a few interesting approaches available with it, which are out of unittest reach.

Writing a test

A single test function can be divided in three steps:

Arrange: create the tests conditions and initialize the environment in which you want run the subject of the test
Act: execute the function you want to test
Assert: verify the function outcome

The hardest part of writing a test is by far the Arrange one.

If you can get away with a limited setup in purely unit tests, as you move one step up in the testing pyramid, and you write integration or service tests, creating the correct conditions can be non trivial.

The risk with this is to put too many assertions in a single test function to “spare” too many tests sharing the same (or similar) test arrangements.

A slightly better solution is to share the arrangement between a few tests by calling a common function, but this can lead to make the tests less readable in the future due to the extra level of indirection.

Parametrize

There is one thing computers are good at: executing repeatedly the same code against different inputs.

What if we apply this to executing tests?

That’s the idea of parametrize:

@pytest.mark.parametrize("test_input,expected", [("3+5", 8), ("2+4", 6), ("6*9", 42)])
def test_eval(test_input, expected):
    assert eval(test_input) == expected

Thanks to how python works, you can treat a function as an input object to another function and let the latter alter the former or execute it with additional parameters (a pattern implemented by python decorators).

Parametrize uses this pattern to call our single test function using the list of inputs provided as arguments to effectively create (and the tests log reflects this) a list of test functions “generated” at runtime sharing the same body.

The bonus of this approach is that you can concisely define a set of input and expectations, yet they are very clearly isolated from the test function body; on the contrary calling shared functions in multiple concrete test functions bury the conditions and expectations in the code making it difficult to quickly isolate them.

A more complex example:

@pytest.mark.asyncio
@pytest.mark.parametrize(
    "headers,status_code",
    [
        (None, 422),
        ({"Authorization": "bla bla"}, 403),
        ({"Authorization": f"Token {LICENSE_API_TOKEN}"}, 200),
    ],
    ids=["no_header", "wrong_auth", "authenticated"],
)
async def test_clear_cache_auth(self, redis, headers, status_code):
    """ Cache is cleared if view is called with proper authentication token, is left intact otherwise. """
    async with httpx.AsyncClient(app=app, base_url="http://testserver") as client:
        cache = await redis(app)

        await cache.set("somekey", "someval")

        url = app.url_path_for("cache_clear")
        response = await client.post(url, headers=headers)
        assert response.status_code == status_code
        check_value = await cache.get("somekey")
        if status_code == 200:
            assert check_value is None
        else:
            assert check_value is not None

Caveats

There are two things which one must be aware when using parametrize:

each test function run is executed in isolation, as if hey were different functions (as you would expect), so the arrange part is also execute multiple times. If it’s time consuming this will make your test suite slower very quickly as you add input combinations
you might to adapt the assertions depending on the expected outcome (if you are testing an email send function you may want to check the email headers of the delivered email the send is successful, which you can’t when testing for send failures): this can get out of hand very quickly and creating a mess of conditions and assertions, I advise against doing more than a single if to distinguish the normal tested function flow and the error handing; take this as signal that you are forcing too many unrelated assertions in a single test and evaluate whether create different test functions or test some of the assertions in other test functions.

Don't do this!:

testdata = [
    (datetime(2001, 12, 12), datetime(2001, 12, 11), timedelta(1)),
    (datetime(2001, 12, 11), datetime(2001, 12, 12), timedelta(-1)),
]

@pytest.mark.parametrize("a,b,expected", testdata)
def test_timedistance_v0(a, b, expected):
    """ This function is intentionally bad. """
    diff = a - b
    if a < datetime(2001, 3, 31):
        a.tzinfo.dst() == 0
    elif a > datetime(2001, 10, 31):
        a.tzinfo.dst() == 0
    else:
        a.tzinfo.dst() == 1
    if b < datetime(2001, 3, 31):
        b.tzinfo.dst() == 0
    elif b > datetime(2001, 10, 31):
        b.tzinfo.dst() == 0
    else:
        b.tzinfo.dst() == 1
    assert diff == expected

Interesting parametrize features

Parametrize allows for very complex scenarios and I encourage you to check the documentation for the details.

Still there are a couple of features worth mentioning

Assign ids to each combination

By default parametrize creates the test function run name by combining the function name and the parameters set (to something like test_timedistance_v0[a0-b0-expected0]) which might be self explanatory. To customize this you can use the ids argument to provide a terser name test_timedistance_v0[forward]

Tests with automatic and manually specified ids:

testdata = [
    (datetime(2001, 12, 12), datetime(2001, 12, 11), timedelta(1)),
    (datetime(2001, 12, 11), datetime(2001, 12, 12), timedelta(-1)),
]


@pytest.mark.parametrize("a,b,expected", testdata)
def test_timedistance_v0(a, b, expected):
    """ Tests with autogenerated ids. """
    diff = a - b
    assert diff == expected


@pytest.mark.parametrize("a,b,expected", testdata, ids=["forward", "backward"])
def test_timedistance_v1(a, b, expected):
    """ Tests with customized ids. """
    diff = a - b
    assert diff == expected

Generated test names:

$ pytest test_time.py --collect-only
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-6.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 8 items

<Module test_time.py>
  <Function test_timedistance_v0[a0-b0-expected0]>
  <Function test_timedistance_v0[a1-b1-expected1]>
  <Function test_timedistance_v1[forward]>
  <Function test_timedistance_v1[backward]>

======================== 4 tests collected in 0.12s ========================

Mark parameters combination

One of the great pytest features are marker which allow to decorate test functions for different behavior (parametrize itself is a marker). You can use marks on a parametrize decorated function as you would normally do, but you can also decorate single parameter set, using the pytest.param function, which you can also use to define a single Id without defined one for each combination

@pytest.mark.parametrize(
    "test_input,expected",
    [
        ("3+5", 8),
        pytest.param("2+4", 6, id="basic_2+4"),
        pytest.param(
            "6*9", 42, marks=[pytest.mark.xfail], id="basic_6*9"
        ),
    ],
)
def test_eval(test_input, expected):
    assert eval(test_input) == expected

Take home

One of the signatures of pytest is to completely embrace the dynamic nature of python, which allows it to work on a “meta” level, by enriching the test behaviors working outside the test functions, thus separating very clearly the boundaries and creating a test suite easier to understand and extended.

Parametrize is a perfect example of the pytest approach and I advise you to experiment with it, as it will greatly improve your test writing experience.

My 2020 Python linting setup: How I Learned to Stop Worrying and Love the automated code formatting

Iacopo Spalletti — Fri, 18 Dec 2020 13:34:40 +0000

Prologue

No matter if a team works using pair programming or by requiring code review before checking in on the main branch, code formatting is a constant source of distraction, when not a source of discussion.

Writing even a few lines of code requires a lot of code style decisions, which, if not agreed in advance, yields a lot of bickering and hard to read code review diff.

The obvious step is to establish team-wide style guide, to avoid arguments and limit messy diff.

But we are developers, we are lazy by nature, and there is little immediate reward in constantly reformatting by hand the code you are writing. You know it matters, but it’s a burden (and an unwise way to spend time) to do it manually.

I have been using a style guide for a very long time, and very early on I started pushing checks in the CI.

But I still felt I was lacking a ton of automation, so I started studying the tools that would near me to the complete automation goal.

I want to share my findings in this quest, that might be useful for others.

Adapting expectations

Automation comes at a cost: you depend on the style rules available in existing linting/formatting tools (unless you want to write one), and you have to balance what you think as “good” code, with what the tools can achieve.

So the first, less pleasing, part of the process is going through the existing tools, and choosing the ones closer to what you want, and then tweaking their output to suit your style; but in the end you will have to accept a few compromises. It took me a while to get along with this (and it’s one of the reasons why it’s been a long process to me).

After a few months of tests with real projects, I think I found a pretty good balance, and I learnt to live with the less pleasing outcomes.

Running the checks

As we depend on tools to handle all the checks, we must decide when to run them.

As everyone in the team is free to use their editor, we decided to keep the checking / formatting configuration in the repository itself, as an easy way to use exactly the same tools without relying on mixed support from each editor.

CI

This made also very trivial to run the same checks in the continuous integration pipeline, as we want to run the checks on every push to ensure code is sane when you open the pull request and avoid he annoying code style comments during the review. Over the years and I have used gitlab-ci, travis, github actions and others. As I wrapped all the checks in [tox environments]((https://github.com/nephila/djangocms-blog/blob/develop/tox.ini) the CI specific details didn’t matter and I ported configurations across CI environments with zero effort.

Pre-commit

Example

But before pushing, you want to make sure everything passes,to avoid wasting CI resources, and your time. For a long time, I relied upon simply running tox locally, which was effective but tedious. And then I decided to give a read to the pre-commit documentation and it made my life so much easier.
Pre-commit create a pre-commit git hook and launch a set of checks on every file staged for commit, before it’s committed: this ensure that committed files respect the configured checks.
This is really game changing, because you can use your coding workflow without any extra step required for formatting and linting: edit, save, add and commit, and all the changes will be validated.
Apart from adding the standalone tools, pre-commit comes with a lot of integrated checks (which partially overlaps with the others, but the added value is totally worth the extra steps) (see config file for the configured checks and pre-commit hooks documentation for the detail).
For projects where python and JavaScript code are in the same repository, JavaScript lint-staged command is also integrated as a pre-commit tool, to run the frontend linting together with the python one.

The checks

Code style and format

This is of course the core of a linting suite, and where the team consensus can be somewhat harder to reach.

Editorconfig

Example
Editorconfig is a specification for basic text formatting (line length, indentation rules etc) which is used by the text editors and IDEs as a way to configure editor automation. It’s not python specific and it only provides basic rules, but it’s very handy as it’s basically transparent, once you enable editorconfig support in your editor.

flake8

Example

Flake8 is the first tool I introduced quite a while ago, and it does quite a good job.

It has a very comprehensive set of rules to check the code for style or logic errors. It covers a lot of non-trivial cases, and it brings a much more consistent code.

In this iteration I added a lot of plugins for quite a strict configuration, even if most of the work is done by black, this still find quite a few quirks worth of being fixed.

Plugin	Description
flake8-broken-line	Report usage of `\` to break new lines, use `()` instead
flake8-bugbear	very opinionated -but very sensible- syntactic and logic checks
flake8-builtins	Report local variables shadowing python builtins
flake8-coding	Used to reject encoding magic comments in files: as codebase is now py3 only, we don’t need those
flake8-commas	Check trailing commas, it has some compatibility issue with black
flake8-comprehensions	Advise about usage of comprehensions and generators to help writing more idiomatic code
flake8-eradicate	Report commented out code: we don’t need it and we can retrieve older versions of the code from git
flake8-quotes	Check quote consistency, somewhat obsolete due to black formatting quotes according to its rules
flake8-tidy-imports	Enforce some rules for module imports

isort

Example
Stop worrying about organising imports has been one of the easiest victory I achieved in my quest for the perfect linting setup.

isort is basically the tool you want to use to organise python imports, it does a great and consistent job, with many different styles to suit any preference, and once you set it up it just runs and do its job. Only occasionally I found weird inconsistencies, but they can be easily fixed. Beware if using with black, as the import formatting style must match the one expected by black.

black

Black is a very much touted tool nowadays, and for good reasons.

It’s fast, it’s easy to use, and it’s really consistent.

It does a great job at formatting the code according to the code style, and I have yet to find an error (as in code that doesn’t run or with black introduced bug).

On the other hand it’s totally opinionated to the point that you barely can configure it. It took me a while to digest this, even more as some of its style decisions look questionable to me. Yet, the output is good enough and the amount of work it lift is so much, that I think it would be stupid not to use it.

pyupgrade

My latest discovery is this tool (which is currently only integrated in pre-commit), which detects and improve the syntax according to the features available in the target python version. Going through its documentation is also very instructive to learn more idiomatic python.

Documentation

One of the most overlooked part of developing the software is the inline documentation (with narrative documentation being the most overlooked part), so a gentle (or not so gentle) nudge toward writing documentation is really important.

Sphinx

The most basic check for your documentation is that it compiles successfully, by adding Sphinx building to the pipeline checks, we ensure the correct syntax. On github you can just configure building documentation on each pull request via readthedocs, which is definitely the best way to test the documentation as it provides a browsable fully rendered version for proofreading and other non automated checks.

On GitLab you can add the relevant job line 13 ensuring the -W flag is used to treat warnings as errors.

pydocstyle

Example
Pydocstyle checks the presence of doc strings at different levels (module, classes, methods, etc), the formatting and its wording style.

Its default configuration is quite aggressive, and it required me quite a few try to balance automatic check stubborness with actual useful documentation.

It’s still somewhat the weakest link in this setup. Maybe because I don’t write enough documentation?🤔

interrogate

Example
It’s a very recent addition and I am still evaluating it (by setting its failure threshold to 0).

On one hand its logic is simpler than pydocstyle: it merely count the number of docstrings-covered “objects” (module, classes, methods, etc).

But this approach provides a more nuanced approach at covering code with documentation, as it allows to avoid adding stub / void doc strings where not needed, or when working on an existing codebase.

My only current concern is its unnecessary failure when an excluded directory does not exist, which leads to more maintenance. I plan to propose a PR to fix this.

Package information

If only I can count the botched pypi uploads due to errors in manifest file or other package meta information (namely correct rst formatting in package long description). While not fundamental, they are a good safeguards for easier applications maintainer life.

check-manifest

Example
Does exactly what its name says, it checks the manifest file against the repo files and report any file not included in manifest declarations.
Even in mature applications, it saved me in a couple of occasions; it has a bit of setup/maintenance work at the beginning as you will have to accurately check which files legitimately are not going in the package file.

pep517

Example
If you are going to release your application as a package, you’d better check that it actually builds. Among other things, pep517 does exactly that.

twine

Example
I use twine to upload built packages to pypi or private registries, but it also provides a command to check the long description syntax, which is really important as pypi is now refusing packages with broken long description, and you don’t want to do multiple releases because of a syntax typo somewhere.

Epilogue

Moving to auto formatted and strictly checked codebase helped me a lot focusing more on the substance of writing code (design and architecture) and less on the formal details.

When working with a team, it also significantly reduced the time for a code review and to get a pull request merged, as one can focus on the implementation by having a cleaner diff and a consistent code.

The journey is far from over, as there is room for improvements in the current setup (starting from checking overlapping between flake8 and black), but overall it is proving a solid base with interesting returns.

TIL - 🧹 unimport linter formatter

Iacopo Spalletti — Sun, 13 Dec 2020 09:44:46 +0000

Jeff Triplett 😷

@webology

✨ Unimport - A linter & formatter for finding & removing unused import statements. - looks good to me. If it works in large codebases as well as it just did on a few smaller projects for me then I'll add it to my toolbelt.

unimport.hakancelik.dev/#/

23:38 PM - 12 Dec 2020

0 10

Thanks to this Jeff Triplets tweet I discovered unimport, which is an interesting tool to detect unused imports (like flake8 would do) and to remove them from the code.

And the cherry on top it's that it can scan the requirements file and detect unused dependencies (which is probably even more important).

Historically this has not been a problem for me (or Nephila team), as flake8 is already well ingrained in our workflow, so we are used to handle code cleanup before committing.

But as I sometimes inherit a large codebase with messy code, and it's awesome how much you can achieve with the right tools.

Currently isort + black + pyupgrade already do a really good job, but adding unimport to the equation is 💥: a much nicer codebase with zero effort

I will surely give it a more in depth test, and I might add to my linting reference toolbox.

Shootout to Hakan Çelik for this package, and its awesome documentation.

Converting a django CMS plugin to a different model

Iacopo Spalletti — Thu, 13 Aug 2020 08:01:26 +0000

Once in a while you might find yourself in the situation in which a django CMS plugin you are using in a project is no longer supported or no longer suited for your goals.
But then, you have hundreds (or thousands) of instances of the old plugin and you need a quick way to migrate all the data to a new one.
Luckily, it's possible to achieve this using a rather simple data migration.

Sample code used referenced below: https://gist.github.com/yakky/31466588da712e25e1f2c200d56d3776

How it works

In order to have a successful migration we must achieve the following:

keep the plugins tree consistent
keep the reference to the plugin valid (eg: the tag embedded in text fields when the plugin is embeddable in a text field)
keep custom data intact (at least the ones supported by the new plugin model)

To achieve this we can use the fact that Django models instances are python classes disconnected from the database until an explicit database update is run.

deceiving Django ORM and the database

So the first step (lines 9-10) is to create an instance of the new plugin class without saving it to the database, then set the id of the new plugin to the one that's being removed; at the end (21-22) we delete the old instance before saving the new one to free any constraint on unique fields.
By keeping the same id, plugin instances embedded in text plugins (whose reference is kept by hardcoding the id in the text body) are moved to the new model.

Migrating tree data

Django CMS plugins are stored in the database using the materialized path algorithm by Django-treebeard, and the plugins hierarchy information is stored in a few fields which neees to be copied verbatim (lines 11-18). This bypass the tree algorithm and we effectively replace a single node in the tree without affecting the tree balancement or triggering the automatic tree rebalance.

Migrating custom data

We now can save the custom model data to the new one. To achieve this we can add the lists of assignments from the field on the old plugin to the one the new plugin: new_plugin.field_name = old_plugin.field_name. If the plugin has any many to many relationship we can copy those as well, after saving our new plugin, using new_plugin.many2many.set(old_plugin.many2many.all()).

Closing

The above snippet has still room for improvements (like using _meta.get_fields to copy the fields data without writing every single field, but should provide you a good starting point to customise them your needs.