Forem: yaythomas

durable functions... AWS Lambda beyond 15mins, with automatic durability and state management!

yaythomas — Mon, 08 Dec 2025 02:29:02 +0000

Eric D Johnson for AWS

Dec 3 '25

AWS Lambda Durable Functions: Build Workflows That Last

#serverless #aws #lambda #durable

Comments

5 min read

Comparison of Python TOML parser libraries

yaythomas — Tue, 14 Dec 2021 16:56:08 +0000

Update February 2022: Since the original article was written below, PEP 680 to add TOML Parsing in the Standard Library was approved.
So tomli, as recommended below, will actually be included in 3.11.

If, like most of us, you're going to have to support older versions of Python for a few years yet, read on. . . this article talks you through your options!

The pypyr automation pipeline task-runner open-source project recently added TOML parsing & writing functionality as a core feature. To this end, I researched the available free & open-source Python TOML parser libraries to figure out which option to use.

If you're interested in a walkthrough of the decision making process, this is documented in an Architecture Decision Record (ADR) you can read here: adr003 toml in the pypyr core

Hopefully sharing my notes helps someone else going through this process to save some time...

TOML is becoming a popular configuration format, and unavoidably so in Python in the shape of pyproject.toml as accepted by PEP518.

Since "best" is a loose term best left to click-bait headlines (hoho, see what I did there? 😉), instead of asking "which TOML parser is the best?" the more sensible question to answer is which TOML library more suits the requirements of your project with the least negatives?

Python TOML libraries at a glance

tomli
- Relatively fast read-only parsing.
- Companion library tomli-w for writing.
tomlkit
- Round-trip white-space/style preserving.
toml
- This was initially vendored in pip itself to deal with pyproject.toml.
- Even so pip has since moved to tomli.
pytoml
- Abandoned by the creator for eminently sensible reasons (interesting read too... https://github.com/avakar/pytoml/issues/15) But let's not get into an argument over whether a shiny new fashion in config formats is in fact doing anything better than the previous fashions in config management...
qtoml
- Still on TOML v0.5.0.

Note that these are the pure Python parsers - there are also others that are basically interop wrappers for fast C++ or Rust libraries.

pytomlpp
- Python wrapper for the toml++ C++ library.
rtoml
- Python wrapper for fast parsing written in Rust.

If performance is your main concern, then the C++/Rust implementations might serve your needs, assuming you're fine with these not being pure Python packages.

Let's investigate the pure Python libraries in greater depth:

tomlkit

Of the available options, only tomlkit supports style-preserving roundtrip parsing. Furthermore, tomlkit was created for the express purpose of handling TOML parsing for the poetry tool. As this is one of the 2 most popular new PEP517 & PEP518 Python build systems there is some comfort to be had in the wide adoption of a very actively used tool that means a greater likelihood of continued maintenance & support, and specifically that pyproject.toml files should parse without surprises.

TOMLKit only lists itself as 1.0.0rc1 compliant. Looking at the TOML spec release history delta of rc1 vs v1, it only looks like clarifications & administrative/documentation updates -there doesn't seem to be anything notable missing or functionally different in rc1 as opposed to v1. It's not impossible that I missed something, though - but given TOMLKit's wide usage via poetry, I would expect obvious out-of-date spec handling to have been noticed by someone somewhere, and I see none such in the issues list.

There is a but... TOMLKit outputs custom types rather than just the standard Python built-ins like dict. Specifically it represents tables with classes like class Table(Item,MutableMapping, dict) or class InlineTable(Item, MutableMapping, dict).

(See here for TOMLkit API types.)

The constructors for these do NOT allow any of these to instantiate like a standard Mapping type does - which may or may not fit your needs, depending on what exactly you're doing. It probably doesn't really matter for most purposes.

toml

The toml library is a largely historical artifact at this point. Not only is it well behind on implementing TOML v1, but also because of a lack of maintenance on extant functionality.

Even pip itself has moved from vendoring toml to tomli. This exodus from toml to tomli includes the very prominent:

(The links are to the issues/PRs discussing the reasons why...)

tomli

tomli, then, seems to be where the Python community in general is coalescing for a "standard" TOML parser. tomli is read-only. For write functionality there is the companion library tomli-w.

tomli is explicitly TOML v1.0 compliant.

tomli is significantly faster than TOMLKit. It does not, however, preserve style/whitespace like tomlkit does. For most use-cases, arguably TOML reading is the important part...

Summary

Use tomlkit if you need to round-trip & preserve style + comments.

Use tomli if you're just after reading a config file or write some output without caring about the formatting too much.

Use the Rust/C++ interop libraries if performance is your main concern and you do not have limitations on using packages that aren't pure Python.

In the case of pypyr, tomli matched the requirements with the least trade-offs. And as a mini-review, it was a joy to use 😄.

If you're interested in seeing a real-world usage example for tomli & tomli-w, you can check it out in action here: https://github.com/pypyr/pypyr/blob/main/pypyr/toml.py

Simplify your automation scripts with a low-code task-runner

yaythomas — Mon, 05 Oct 2020 13:20:12 +0000

pypyr automation task-runner is a free open-source tool that lets you script sequential task workflow steps in a pipeline using yaml. You get conditional execution, loops, error handling & retries without writing custom code.

pypyr runs pipelines from a clear straightforward CLI. You can also use this as a library programatically from your own code - the pypyr API to run a pipeline is undemanding and clearly documented!

Why should you use a task-runner?

You know how your automation scripts usually start off with a couple of innocent lines of bash, or maybe python?

And then they get out of hand?

And then before you know it you've got a big clumsy monster that you really hope doesn't break because you're too scared to touch it?

And then you have to code & test the repetitive not-all-that-interesting plumbing of:

sequencing tasks
parse & validate cli input arguments
parse config files, write output files
handling errors
retrying commands on error
running a group of commands in a loop
invoking the cli from python or invoking python from the cli

Streamline automation with a task-runner

pypyr task-runner helps with all that. With pypyr you create your automation workflows in a human-friendly yaml pipeline.

Add loops, error-handling & automatic retries to your command & program sequences, without having to write & troubleshoot code to do so.
Create your own cli app, with your own custom command line arguments, without having to do all that arg parsing & validation yourself.
Control-of-flow where you conditionally execute or skip steps based upon input expressions & switches.
Easily work with config files like json, yaml or plain old text to parse & substitute values.
No-code to low-code way of sequencing commands & programs, while still having the power to drop into full-on Python if you need to express something more complicated.

Add automation to existing utilities without writing code

Here is an example pipeline to retry a curl command up to four times until it's successful, at 0.5 second intervals:

The reason I first wrote pypyr, is because when I'm writing automation scripts, I find myself jumping between bash/shell and a more advanced language like Python to get things done efficiently.

For example. . . sometimes a quick sed from the cli is quicker to implement, sometimes more advanced string parsing in Python is necessary - sometimes both things are necessary at different parts in the same automation sequence.

Some tools you need to invoke to automate your tasks only have CLIs, other tools only have APIs. . .

The famous Unix philosophy of building programs that do one thing and do them well has given us a lot of battle-tested amazing utilities that have been around for decades (think curl, grep, awk, sed. . .)

Why not use these, rather than reinvent the wheel? So the point of pypyr is to make it easier for you to stitch together those awesome utilities, do some of your own scripting in between them, and generally automate it all without the usual repetitive pain points.

Why use a task-runner if you could just code it all yourself instead?

The obvious hard-core developer response is:

yeah, I just code that myself, why bother with this?

And yes, of course, you totally can code all of the surrounding, boring, repetitive, not-all-that-interesting bits of automation code for retries and err handling etc. yourself. . .

But I'm sure you'd rather keep yourself busy with doing cool new algorithms or functional code, rather than get mired down in automation tasks!

Free & Open-source

The source code is here: https://github.com/pypyr/pypyr/.

If you want to contribute, even just a GitHub star is useful!

I wrote the 1st code for this years ago when I found myself having to solve the same sort of problems each time I wrote some sort of automation script. . . Over the years it's grown as I've found more & more uses for it and expanded upon it, and because it was open-source from day one pypyr has been blessed with great enhancements from superb contributors over the years who brought ideas, enhancements and fixes that I wouldn't even have thought of. Big shout out to you all!

The new pypyr technical documentation website source-code itself is also open-source, and packaged as a re-usable Hugo static site generator theme.

100% Test Coverage

From the beginning I've made an effort to keep 100% test coverage over both unit & integration tests, and this (although annoying to keep to that sometimes. . .) has been invaluable in preventing regressions & providing steady stability of the tool as it evolved, but let me not break my arm patting myself on the back here. . . this implementation detail is probably not all that interesting unless you're into code.

Feedback & Help

Feedback, comments and/or ideas to make things better on pypyr itself and also the new pypyr documentation website are very welcome. . .

Also, if you have questions, feel very free to ask, I'm here to help!