Forem: Chris White

The Road To Kubernetes: How Older Technologies Add Up

Chris White — Mon, 05 Feb 2024 17:16:03 +0000

Virtualization
Resource Isolation
Resource Control
Containers
Docker Joins The Stage
Docker Growth
Introducing Kubernetes
Kubernetes And The Cloud
Component Separation
The Future?

At some point in a DevOps engineer's career they will hear about Kubernetes. It's not a matter of if, but when. With containers being the new hot thing and Kubernetes orchestrating their deployment it's easy to see the reason for its popularity. Even so, Kubernetes is an accumulation of previous technologies. In this article I'd like to discuss the previous way of handling similar application hosting solutions and show how they lead up to a lot of what Kubernetes offers today.

Virtualization

Before containers were popular, virtualization was the way you could split up a single server into multiple "containers". At first they relied on paravirtualization which is what the first EC2 instances used. Virtual Machines would interface to the operating system via some form of middle man which would translate the guest OS's calls to what the host OS would expect. Given the requirement of such a conversion these virtual machines wouldn't be able to keep up with running software straight on the hardware itself. While this wasn't a big issue for some solutions it did mean that adopting it widely where applications cared about performance was a difficult endeavour.

Hardware virtualization would start to gain traction through the usage of various CPU related technologies specifically to improve performance. Technologies such as KVM were made to further improve integration with this new technology. However, there were still issues with some hardware interaction that still made it not match up to bare metal. Another alternative was to simply use the same hardware, but isolate the application itself.

Resource Isolation

Resource isolation primarily started off with isolating how a process could operate. The most popular example of this was chroot (1979). More specialized examples of this included FreeBSD jails (2000) and Solaris Zones (2005) which still kept some of the virtualization aspects. Now chroot, which stands for change root, allowed a process to have their root mount point changed effectively isolating it on a filesystem level.

For these applications to properly run on a Linux system they need access to special paths which provide system information through a filesystem interface. This includes procfs, devpts, and sysfs. These allowed for different mount options for the chroot than the host system they were running on. More simple variants could be done with bind mounts. The underlying application also needed access to its dynamically linked libraries to run. Even static binaries still need access to a c library implementation (ex. glibc) and system calls making setup more difficult if you wanted best performance.

From a security standpoint this setup was beneficial for webservers who only needed access to their respective HTML. CSS, JS, and any dynamic content generation files. If a basic malicious actor obtained access to a local shell, they would only be able to see files inside the chroot. That's not to say that chroot's are some kind of unbreakable fortress. Another issue was that there wasn't a very easy packaging solution which would replicate the ease of use of a virtual machine image. This complexity only increase as more applications are added to a server. Out of the box chroots also don't provide any kind of resource quota mechanism as you would expect from modern day containers.

Resource Control

Several technologies were introduced to the Linux kernel to help address some of these issues. namespaces were introduced into the Linux kernel in 2002 for more fine grained isolation. They allowed for scoping resource availability on a process level. You could have one process see one set of resources, and another process see a different set of resources. Next is quotas on resources which has handled by cgroups, released in 2007. They provide a special filesystem mount to configure settings such as what CPUs a process can utilize, how much memory/cpu it can use, and how much network bandwidth it can use. The final set of controls are capabilities, introduced in 1999. While a foundation for process isolation its original purpose was for restrictions on privileged user accounts. Despite its early introductio, the full set of capabilities available today is something that was an accumulation of additions over several kernel releases. While these are powerful technologies, they are also fairly complicated to manage in a scalable manner. They also had the same issue with chroots in that there wasn't a very easy way to package everything together.

Containers

While Docker helped drive container adoption, it was LXC which provided the base foundation to make the system work. In fact early versions of Docker utilized it as a backend. It provided the packaging of kernel level features such as cgroups and namespaces which were used for container like isolation. Despite it's features LXC still lacked the modern day interface you'd expect to easily deploy containers. There was also a lack of major cloud managed service providers to help with overall adoption.

Docker Joins The Stage

The initial release of Docker was on March 20th 2013. Important to note is that containers are a more of a concept, and Docker is an implementation. What Docker brought to the table was a container image hosting service as well as software to orchestrate the more difficult parts of the container setup process. Docker also provided a very simplistic image definition: the Dockerfile. Compared to what needed to be done for virtual machine style builds Dockerfile builds simplified the process and used an incremental diff patching feature which enabled more modular organization of images. Images hosted on Docker's platform could be utilized in this process allowing for vendor solution tie-in. Having a business backing it also helped with corporate adoption of containers.

Docker Growth

Despite this great set of features Docker still would take some years to become the popular solution it was today. This was attributed to the security and scalability concerns of the original iteration. In 2014 Docker Compose was released as a solution to turning Docker containers into a more "project" like layout. This further helped organization of containers along with the incremental diff functionality. This was further improved via the introduction of Docker Swarm which was released as part of Docker 1.12 in 2016. This allowed a more scalable cluster setup for Docker containers.

Introducing Kubernetes

Kubernetes was first released on September 9, 2014. This release timeline is part of what helped it gain a foothold over Docker Swarm. It was an open source version of an internal Google project. Features of container orchestration were presented in a more modular fashion along with scaling functionality. You can chose how your networking stack works, your load balancing, container runtime, and filesystem interfaces. Availability of an API allowed for more programmatic interactions with orchestration, making it tie in very well with CI/CD solutions. However, the big issue it has is complexity of setup. Putting together a Kubernetes cluster with basic functionality is certainly no easy feat.

Kubernetes And The Cloud

Due to the amount of effort required not to just setup, but also migrate to Kubernetes, original enterprise adaptors would rely on the Google Cloud Kubernetes Engine to abstract the effort. The product was released shortly after Kubernetes itself, with the first release on November 4th 2014. Unfortunately, due to Google Cloud's 2 year gap with AWS's public release this still made Kubernetes adoption on a wider scale more difficult. AWS' first step into the container space was interestingly enough Elastic Container Service or ECS in 2015. While not bound with Kubernetes specifically the ease of deployment made it an interesting competitor for more simplistic container workloads. Google's best alternative to it, Google Cloud Run would not see an available release until 2018.

AWS would then release Elastic Kubernetes Services or EKS in 2018. Azure tagged on slightly afterwards with their own Azure Kubernetes Service or AKS in the same month. With the 3 major cloud providers all having a managed service Kubernetes started to see a large spike in adoption over the years. So much that a majority of DevOps job descriptions now have it as a technology stack component.

Component Separation

Kubernetes on the backend used to utilize docker for much of its container runtime solutions. One of the modular features of Kubernetes is the ability to utilize a Container Runtime Interface or CRI. The problem was that Docker didn't really meet the spec properly and they had to maintain a shim to translate properly. Instead users could utilize the popular containerd or cri-o runtimes. These follow the Open Container Initiative or OCI's guidelines on container formats.

Projects central to the container experience began to join the Cloud Native Computing Foundation. While it supports many popular open source DevOps related projects, much of the governing board and technical oversight committee are corporate oriented. Kubernetes itself also did a migration of their download site from Google to a more open one which would allow for adding additional mirrors.

The Future?

While Kubernetes is in what I consider to be its prime now, like all technologies there's certainly room for a new tech to take its place. Especially true given the complexity of it even when run in a managed environment. A bump in the right direction could certainly be made by having managed services with specific configurations for specific use cases to be available. Centralization of monitoring/security/performance data along with basic administration would also help it substantially.

If you like what you see here I'm currently available for remote full time work. Check my profile for links.

Understanding CVE Reports

Chris White — Sat, 06 Jan 2024 02:18:15 +0000

What Is A CVE
CVE Format
CVE Analysis
- Is The Affected Software Present
- Severity
- Exploit Vector
- Fix Version
- Embargo
- Validation
Having A Reporting Program
Conclusion

CVEs (Common Vulnerabilities and Exposures) are an official collection of recognized security issues. Knowing these is valuable not just for security, but for other teams such as DevOps and development. A major problem is that some people don't know how to handle a CVE properly. This can lead to unnecessary panic and a loss of valuable company time. In this article I'll be going over what a CVE is and how to analyze one.

What Is A CVE

The concept of CVE was created back in 1999 as a way to centralize vulnerability details. While there is a central governing organization, MITRE, actual vulnerability reporting is often handled by other parties. These are known as CVE Number Authorities or CNAs. CNAs span a number of scopes which indicate what CVEs they can report against. For example, Adobe is authorized to publish a CVE for one of it's products, but would not be able to publish a CVE for the Python programming language. Some notable CNA examples:

Adobe
Apple
Python Foundation
Oracle

There are also root CNAs which handle organization of specific scopes of CNAs. Google is one example of a root CNA which covers organization of CVEs related to non Android and Chrome Google products.

CVE Format

CVEs are currently published based off a JSON format specification. Here's an example of an entry for CVE-2023-0038 hosted on the CVE Database in GitHub:

{
    "data_version": "4.0",
    "data_type": "CVE",
    "data_format": "MITRE",
    "CVE_data_meta": {
        "ID": "CVE-2023-0038",
        "ASSIGNER": "",
        "STATE": "PUBLIC"
    },
    "description": {
        "description_data": [
            {
                "lang": "eng",
                "value": "The \"Survey Maker \u2013 Best WordPress Survey Plugin\" plugin for WordPress is vulnerable to Stored Cross-Site Scripting via survey answers in versions up to, and including, 3.1.3 due to insufficient input sanitization and output escaping. This makes it possible for unauthenticated attackers to inject arbitrary web scripts when submitting quizzes that will execute whenever a user accesses the submissions page."
            }
        ]
    },
    "problemtype": {
        "problemtype_data": [
            {
                "description": [
                    {
                        "lang": "eng",
                        "value": "CWE-79 Improper Neutralization of Input During Web Page Generation ('Cross-site Scripting')"
                    }
                ]
            }
        ]
    },
    "affects": {
        "vendor": {
            "vendor_data": [
                {
                    "vendor_name": "ays-pro",
                    "product": {
                        "product_data": [
                            {
                                "product_name": "Survey Maker \u2013 Best WordPress Survey Plugin",
                                "version": {
                                    "version_data": [
                                        {
                                            "version_value": "*",
                                            "version_affected": "="
                                        }
                                    ]
                                }
                            }
                        ]
                    }
                }
            ]
        }
    },
    "references": {
        "reference_data": [
            {
                "url": "https://www.wordfence.com/threat-intel/vulnerabilities/id/a2a58fab-d4a3-4333-8495-e094ed85bb61",
                "refsource": "MISC",
                "name": "https://www.wordfence.com/threat-intel/vulnerabilities/id/a2a58fab-d4a3-4333-8495-e094ed85bb61"
            },
            {
                "url": "https://plugins.trac.wordpress.org/browser/survey-maker/tags/3.1.4/public/partials/class-survey-maker-submissions-summary-shortcode.php?rev=2839688#L311",
                "refsource": "MISC",
                "name": "https://plugins.trac.wordpress.org/browser/survey-maker/tags/3.1.4/public/partials/class-survey-maker-submissions-summary-shortcode.php?rev=2839688#L311"
            }
        ]
    },
    "credits": [
        {
            "lang": "en",
            "value": "Chloe Chamberland"
        }
    ],
    "impact": {
        "cvss": [
            {
                "version": "3.1",
                "vectorString": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:C/C:L/I:L/A:N",
                "baseScore": 7.2,
                "baseSeverity": "HIGH"
            }
        ]
    }
}

The basic breakdown is:

A description of the CVE
Type of vulnerability (cross site scripting)
What is affected
References (such as vendor postings and bug reports)
Credit for report
Overall impact

These are sufficient data points to have a basic idea of if one is affected by a vulnerability.

CVE Analysis

Since we have the information at hand on what a vulnerability is about it's time to analyze it for potential impact. One thing to keep in mind is that each CVE has a set of circumstances to be considered vulnerable. That means specific setups may not be impacted by it, and that upgrades should be handled in an informed manner to reduce unnecessary upgrades which bring unnecessary risk with it.

Is The Affected Software Present

First off is seeing if the software in question is part of your infrastructure. Server software such as nginx are generally easier to tell. In corporate environments there may be a vulnerability scanning solution in place (or something such as dependabot). There could also be some kind of software inventory system in place as well. If you're in a corporate environment and don't have a system in place, I highly recommend looking into it.

Severity

Next to consider is the severity, which is an accumulated score system known as CVSS (Common Vulnerability Scoring System). Calculation of this is based on a number of properties of the exploit, some including:

Is a proof of concept exploit available?
What level of permissions are needed for the exploit?
Is an official fix available?
Is network access required?

and so forth. NIST (National Institute of Standards and Technology) has a calculator with each of the properties explained. The resulting score will indicate how severe the exploit is. For example on the previous wordpress plugin CVE:

    "impact": {
        "cvss": [
            {
                "version": "3.1",
                "vectorString": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:C/C:L/I:L/A:N",
                "baseScore": 7.2,
                "baseSeverity": "HIGH"
            }
        ]
    }

The attack vector is network based, complexity is low, no permissions are required, and there is no user interaction required. This puts it at a higher severity due to this. Even so the resulting impact is fairly low to critical components such as system availability and integrity. Most security teams will want high and above vulnerabilities handled in a very short time frame. Lower severity indicates the ability to pull off the exploit might not be feasible or apply to your particular configuration.

Exploit Vector

Now that severity has been analyzed the next step is to look into how this particular vulnerability is exploited. This is important in cases of highly configurable software such as application servers where features can be enabled/disabled at will. Lets take for example CVE-2022-41741's description text:

NGINX Open Source before versions 1.23.2 and 1.22.1, NGINX Open Source Subscription before versions R2 P1 and R1 P1, and NGINX Plus before versions R27 P1 and R26 P1 have a vulnerability in the module ngx_http_mp4_module that might allow a local attacker to corrupt NGINX worker memory, resulting in its termination or potential other impact using a specially crafted audio or video file. The issue affects only NGINX products that are built with the ngx_http_mp4_module, when the mp4 directive is used in the configuration file. Further, the attack is possible only if an attacker can trigger processing of a specially crafted audio or video file with the module ngx_http_mp4_module.

In this case you'll only be affected if you're using the mp4 module in nginx and the system is allowed to process user input media files. That means if you aren't working with the plugin there is no rush to implement a fix. There also may be environmental aspects to a vulnerability. CVE-2023-5528 for example only affects kubernetes nodes running on Windows systems. Ensuring you understand exploit vectors can help avoid unnecessary prioritization that could put business value add tasks on hold.

Fix Version

This is one of the areas where CVE scanning may show its true colors. Taking a CVE at face value, you might think you simply update to the latest version and then it's done. It's not that simple however. The version you're currently on and the version to upgrade two may contain changes that are not related to the vulnerability at all. This means introducing additional risk into systems in case another non-security bug is present. Due to this it's not uncommon for vendors to implement backports. This is where they take the patch and make it work for a previous version of the package. This helps reduce the introduction of risk into systems. For example, if we look at CVE-2023-46724:

{
    "product_name": "squid",
    "version": {
        "version_data": [
            {
                "version_affected": "=",
                "version_value": ">= 3.3.0.1, < 6.4"
            }
        ]
    }
}

The affected versions are greater than 3.3.0.1 and less than 6.4. However, if we look at the RedHat security advisory you'll notice that the fixed package is "squid:4" or more specifically squid-4.15-7.module+el8.9.0+20975+25f17541.5.x86_64.rpm. Due to RedHat Linux being targeted to slower moving enterprise customers, it's within their business interest to backport instead of updating to the mentioned version 6.4 in the original report. This is why when analyzing a CVE you need to understand where your source of the package is, for example:

Your distribution
Part of a container or virtual machine image
Part of a language's package manager
Manual installation via automation such as AWS Systems Manager or Ansible

It is important to note that you may not always have the luxury of a backport available. In such cases you can attempt to do it yourself, or just deal with the added risk of unrelated changes (going over the changelog of a package is a good idea).

Embargo

This is a special case for CVEs which have an extremely large number of affected systems and require coordinated effort to push a release out. OpenSSL is where you'd be likely to see an embargo occur (this happened somewhat recently with CVE-2022-3786). This gives time for vendors to upgrade systems / packages (such as Microsoft for example) and allows companies to ensure proper staffing to deal with the issue. The primary purpose of all of this is to prevent the possibility of exploits out in the wild causing a war room situation. That's not to say an exploit won't show up, but the less the chance the better.

Validation

Now assuming you are affected it's time to take action. I will note that some extremely serious vulnerabilities (such as embargo'ed ones) you may not have the luxury of full validation due to the time sensitive nature of it. How you achieve a package update will depend on your particular setup and the source of your packages. Depending on the impact of updating packages it may require a scheduled maintenance window. Before deployment though it's recommended to do some validation. If the report has a proof of concept it's possible to use this to validate that your upgrade will work. You should only use this to validate if the PoC has no destructive operations such as deleting files or DDoS-ing a server. I recommend doing an upgrade and run of the PoC ON A NON PRODUCTION DEV/TEST SYSTEM. If you're hosting on a cloud provider such as AWS you might need permission ahead of time depending on how the PoC operates.

You'll also want to do system integrity validation with the new version. This is where you do the upgrade, then run your test suites (or cry because you don't have any) to ensure nothing will break. A broken server can sometimes be seen as worse as a vulnerable one. In some cases you may need to report back to the original bug report where the issue was found. Once everything looks good you can look towards moving it to productions. Blue/Green deployment is another method of helping ensure everything is good by doing a test deployment with validation to ensure critical applications are available.

Having A Reporting Program

While I've covered how to analyze CVEs there are cases where software you work on may be the target of a vulnerability. The first step is having a dedicated channel for reporting potential security vulnerabilities. It should be highly monitored and if possible initial response made within the same business day. Always keep in communication with the reporter. Failure to do so mean they could potentially release their vulnerability findings (with a potential exploit) before you have a time to fix it. If you expect that security reports may be somewhat more common it might be a good idea to become a CNA and report CVEs on your own.

Some organizations even go so far as to provide financial incentive to report security issues through bug bounty programs. The Department Of Homeland Security has a bug bounty program for example. In some cases you may see amounts anywhere from $10,000 - $20,000 depending on the impact of the security vulnerability. Whether or not an organization needs a bug bounty program very much will depend on how high value of a target it is (a mom and pop shop's online store is probably not a good case for one).

Conclusion

I hope this article has provided insight on how CVEs are formatted, and what information you can get out of them. Being able to understand the potential impact of a CVE can go a long way in ensuring the relevant fixes are properly prioritized, and it's not putting crucial project work at risk unnecessarily. I highly recommend understanding how CVEs work even if you're not in a security specific role. It can really help if you need to implement the fixes yourself.

Different Levels of Project Documentation

Chris White — Fri, 29 Dec 2023 22:46:56 +0000

Naming Convention Documentation
Typing Documentation
Comment Documentation
Project README
Internals Documentation
Process Documentation
General Usage Documentation
API Documentation
Advanced Documentation
External Documentation
Hosting Documentation
Conclusion

Documentation is extremely valuable for both open source and internal company projects. With that in mind the question arises on what exactly needs to be documented. While code docs generated from inline formats (ex. python doc strings) may be what is considered to be documentation for developers, there's more to it than that. In this article I'll be breaking down different kinds of documentation and what gives them value to their target audience. Layout wise I'll start at the lowest level and move upwards.

Naming Convention Documentation

First off is the naming of variables and functions which drive the code. A common mistake for newer developers is having code with a lot of inline comments. Much of these verbose comments could be addressed by simply having a naming convention which provides clarity on what the purpose of the function or variable is. As an example:

isAddressPOBox()
Address.isPOBox()
ec2_boto_client

The first and second example shows how naming can change between context. Meanwhile ec2_boto_client could be useful if you're dealing with multiple boto clients connecting to the AWS API. Something to note here is that isAddressPOBox() could technically just be isPOBox but I find that the verbose version also gives context on the input. I would say that you want to get into the habit of doing this for even personal pet projects. It's also very valuable in team environments to make code easier to read (and potentially faster PRs).

Typing Documentation

Typing is useful for giving hints on function input and output. While this is default for statically typed languages, some dynamic languages such as python may support type hinting features for this particular purpose:

def average_numbers(numbers: list[int]) -> float:

In this example I know I need to provide a list of integers as input and will obtain a float as output. You will want to be careful on how you organize custom types. A lack of organization can lead to a lot of back and forth on source code reading. Another nice benefit of typing is that it can be used by many IDEs to enhance linting features by seeing if inputs and outputs match the requested types. This is useful for cases where your project is a dependency library as well as working on team environments.

Comment Documentation

Documentation through code comments can be useful in cases where code may rely on complex algorithms or when something is done in an unusual manner. I find most of the code I comment on tends to be related to cryptographic hashing:

).sign(self._key_object, hashes.SHA256())
# SHA256 was chosen here instead of SHA512 as a compromise between decrypt performance and
# hash security

In this example I'm explaining the reasoning why I'm using SHA256 to sign a cert when the more powerful SHA512 hash exists. TODO comments are a specialized form of comment documentation which allows developers to keep track of code improvements quickly, which can be transitioned to tickets/issues/tasks later on. I will warn on being careful on your ratio of comments to code. At a certain point the actual code becomes difficult to follow and could be a sign that your naming convention/typing might have issues.

## Application Interface Documentation

This style of documentation is often what developers consider to be "code documentation". Users may reference it for libraries to figure out what functions need to be called. Developers may use it to understand the code base. Such documentation is often found after a function/method signature to document inputs, outputs, and a basic summary of what the code does:

 def average_numbers(numbers: list[int]) -> float:
    """Average a list of numbers

    :param numbers: The list of numbers to average
    :type numbers: list[int]

    :return: The average of the numbers
    :rtype: float
    """
    return np.average(numbers)

Important to note is that the format of application interface documentation will vary depending on the programming language and documentation solution. If you find that files become too verbose due to application interface documentation it might be a sign of functions trying to do too much or a need for separation refactoring.

Project README

I would consider this the starting point of figuring out basic information for a project. This includes items such as:

Why the project was created
Status of the project
Project requirements
How to install the project (or a link to an install guide for more complex projects)
Basic usage / getting started
Security policy
Links to further documentation
Contact information (filing bug reports, etc.)

A well laid out README can often be a great way to get project adoption as it brings an assumption that the rest of the project is well documented and users will find information they need easily.

Internals Documentation

This somewhat ties in with application interface documentation. It's general used to supplement such documentation and primarily geared towards developers, power users, and potential contributors to a project. A few examples of internals documentation:

In general this type of documentation tends to be useful at larger scale products with wide user adoption. Understanding internals of a project can also help in finding potential performance optimizations or conditions that might cause bugs to occur.

Process Documentation

This is used to describe various processes a project might have in dealing with the codebase. Some examples of this include:

How to contribute to a project
Setting up a development environment
Project code of conduct (in particular enforcement of it)
Bug reporting
Security reporting guidelines (different from basic bug reporting as it tends to be done in a more private manner to avoid early exploits in the wild)
Release process

It's essentially how users might interact with a project versus how to use the actual project itself. This category of documentation is primarily oriented towards projects with a high contribution rate.

General Usage Documentation

General usage documentation is geared towards end users of the project. This may replace parts of the README documentation if certain usage details require a substantial explanation. General usage documentation often touches on the following components:

Downloading the project (source code/package/installer)
Installing the project
Configuring the project
Validating the project (generally in the form of a quick start guide)

Application interface documentation may be included here as well if the project in question is primarily used as a dependency. Smaller projects may combine download, installation, and configuration steps. Larger projects may need them broken out to handle environment variations such as operating systems, package managers, service management, database engines, etc.

API Documentation

Often used for cases where a project exposes a REST or other type of API service. Open API is a popular method of documenting such API services. It can also be used along side tools such as Swagger Codegen to produce boilerplate code for API interaction / testing purposes. There may also be support files for popular API testing tools such as Postman or Insomnia. This makes it easier at a glance to see what data is coming back from a call so the user knows how to handle parsing the data.

Advanced Documentation

General usage documentation works well for covering the standard user needs for the project. Advanced documentation enhances this by showing more advanced setups which are useful but not what you would expect from an out of the box setup. Such documentation could touch on manual configurations that are normally set to developer recommended settings to cover a majority of use cases. Finally there are cases where subject matter expertise is required such as BGB network integration with Kubernetes networking providers. Such documentation is generally recommended for more active projects where the developer is more aware of project use cases.

External Documentation

This may also be considered "community documentation". Someone writes a blog post on a project, maybe a key note video is posted, essentially anything that isn't directly hosted on core project infrastructure. It may also point to community resources such such as forums, slack channels, and social media sites. One word of caution is that I recommend avoiding slack exclusive documentation for a project as there is a barrier of entry in accessing the information (not to mention separation of concerns).

Hosting Documentation

Once you have all the documentation worked out a place to host it will be necessary. Some documentation generation may have ties in with specific hosting sites. Read The Docs' support for Sphinx and other documentation tools is one example. GitHub pages can be useful for GitHub hosted projects as it integrates well with GitHub Actions CI/CD deployments.

If you want to self host and don't mind paying a bit a combination of Route53 (unless you host DNS elsewhere), S3, and CloudFront on AWS can provide a pretty reliable and cost effective site hosting. It's also a great solution if you're working on an internal company project with architecture hosted on AWS. It's also nice in compliance environments since you can implement strict access controls when necessary which can integrate with company SSO and other IAM components.

Wikis are another solution if your contributors are more comfortable with them. It also helps prevent the issue of "documentation updates caused a big CI run" that can catch some projects off guard. For GitHub users there is a fairly minimal GitHub Wiki feature for projects that support it. This also works on the enterprise versions if you're doing an internally hosted project.

Conclusion

As someone who has been a proponent of documentation for much of my career, I hope this provides some insight on the depth of documentation you can have for a project. I will say that some types of documentation tend to show their true value at certain project growth stages. Trying to do advanced usage documentation is not ideal when your project is starting out and you don't have enough usage information. With that said I recommend looking over each documentation type and evaluating if it would help with project adoption / contributions.

Implementing Quality Checks In Your Git Workflow With Hooks and pre-commit

Chris White — Wed, 13 Dec 2023 15:45:40 +0000

Hooks and Webhooks
Git Hook Basics
Practical Git Hook Usage
Git Hook Automation With pre-commit
Working With Contributed pre-commit Code
File Filtering
Conclusion

Git is a powerful tool to enable developers to organize and share their code. One of the more interesting features is hooks. This allows for execution of scripts at certain points in the git workflow. In this article I'll be showcasing one of the simple git workflow events: pre-commit. I'll also be introducing a tool to help with automation of it, the aptly named pre-commit. Note that the code used for this is the final step of my python beginners series and can be found in a code repository in GitHub. Simply click on the green "Code" button and select "Download ZIP". Then extract it into your preferred directory of choice.

Hooks and Webhooks

To clear up any potential confusion, Git hooks and GitHub webhooks are different entities (though GitHub webhooks most likely is powered by git hooks). Git hooks are specific to the git software package. GitHub webhooks are a feature of the GitHub platform that pushes JSON payloads based on various GitHub related events. The same goes for similar source control service sites such as GitLab

Git Hook Basics

A git hook is essentially a script that is executed at a certain point in the git workflow. The basic rules of hooks are:

Set as executable
Strip the extension (with the exception of certain windows extensions such as .exe)
Located in a .git/hooks directory under the repository parent directory

To see how this looks download the code mentioned in the introduction paragraph and make sure there's no .git directory in it (remove it if there is). Then run git init to initialize the repository:

$ git init
$ git branch -m main

The second command ensures the default branch is main, which is fairly common for new repositories on major code hosting sites. After the git repository has been initialized it's time to look at the contents of .git/hooks:

$ cd .git/hooks
$ ls -1
applypatch-msg.sample
commit-msg.sample
fsmonitor-watchman.sample
post-update.sample
pre-applypatch.sample
pre-commit.sample
pre-merge-commit.sample
prepare-commit-msg.sample
pre-push.sample
pre-rebase.sample
pre-receive.sample
push-to-checkout.sample
update.sample

Each of these has examples. To make them work we'll need to make a copy with .sample removed and ensure it's executable. Let's take a look at the pre-commit.sample contents:

#!/bin/sh
#
# An example hook script to verify what is about to be committed.
# Called by "git commit" with no arguments.  The hook should
# exit with non-zero status after issuing an appropriate message if
# it wants to stop the commit.
#
# To enable this hook, rename this file to "pre-commit".

if git rev-parse --verify HEAD >/dev/null 2>&1
then
        against=HEAD
else
        # Initial commit: diff against an empty tree object
        against=$(git hash-object -t tree /dev/null)
fi

# If you want to allow non-ASCII filenames set this variable to true.
allownonascii=$(git config --type=bool hooks.allownonascii)

# Redirect output to stderr.
exec 1>&2

# Cross platform projects tend to avoid non-ASCII filenames; prevent
# them from being added to the repository. We exploit the fact that the
# printable range starts at the space character and ends with tilde.
if [ "$allownonascii" != "true" ] &&
        # Note that the use of brackets around a tr range is ok here, (it's
        # even required, for portability to Solaris 10's /usr/bin/tr), since
        # the square bracket bytes happen to fall in the designated range.
        test $(git diff --cached --name-only --diff-filter=A -z $against |
          LC_ALL=C tr -d '[ -~]\0' | wc -c) != 0
then
        cat <<\EOF
Error: Attempt to add a non-ASCII file name.

This can cause problems if you want to work with people on other platforms.

To be portable it is advisable to rename the file.

If you know what you are doing you can disable this check using:

  git config hooks.allownonascii true
EOF
        exit 1
fi

# If there are whitespace errors, print the offending file names and fail.
exec git diff-index --check --cached $against --

So this will do a few basic checks against file names and whitespace issues. As one of the comments mentions To enable this hook, rename this file to "pre-commit". We can then just run git commit afterwards to run it:

$ cp pre-commit.sample pre-commit
$ cd ../../
$ git commit

At this point nothing happened as nothing was added. I'll go ahead and add a file with a Japanese name to see what happens:

$ touch テスト
$ git add テスト
$ git commit
$ git commit
Error: Attempt to add a non-ASCII file name.

This can cause problems if you want to work with people on other platforms.

To be portable it is advisable to rename the file.

If you know what you are doing you can disable this check using:

  git config hooks.allownonascii true

As you can see our hook is working and is preventing the non ASCII named file from being committed. I'll go ahead and cleanup the test:

$ git reset テスト
$ rm テスト

Practical Git Hook Usage

So we've seen a very basic example of how hooks work. Now the more practical example would be to use it to run tests on code before committing. Note that I don't recommend this if your project is in the prototyping phase where you may be doing many commits and may not have tests setup due to the frequency of code architecture changes. In this case the project in question has tox setup which runs various python linting and tests. Before running the examples you'll need to ensure pdm is installed and running under python 3.11. Instructions for this can be found in my pdm tutorial, or you can install it on your own. Once installed we'll make sure the necessary packages are available for tox to work with:

$ pdm install

Then I'll edit the .git/hooks/pre-commit file to contain the following:

#!/bin/sh
pdm run tox

if [ $? -ne 0 ]; then
  echo "tox checks failed" >&2
  exit 1
fi

Now when git commit runs:

$ git commit
lint: install_deps> pdm sync --no-self --group testing --group lint
lint: commands[0]> flake8

tox is run to initiate all the checks. As is the code should be in working condition so you'll see a commit screen after the tox run. Go ahead and close it out without putting a message to abort the comment. When there's an issue with something (I went ahead and commented out one of the imports):

$ git commit
lint: install_deps> pdm sync --no-self --group testing --group lint
lint: commands[0]> flake8
<snip>
  lint: FAIL code 1 (4.34=setup[3.69]+cmd[0.65] seconds)
  test: FAIL code 1 (11.80=setup[10.09]+cmd[1.71] seconds)
  docs: OK (14.44=setup[11.79]+cmd[2.65] seconds)
  evaluation failed :( (30.79 seconds)
tox checks failed

A message at the end shows that tox has failed to run and no commit screen is displayed.

Git Hook Automation With pre-commit

Given how useful checking code quality is before committing, there's actually a framework around git hooks called pre-commit. It's somewhat like a mini-GitHub Actions that can be used to run various commands during the pre-commit phase. This is where you normally want most CI/CD like checks. Before continuing you'll want to remove the existing hook or you'll end up having duplicated tool runs. We'll also go ahead and add all the files in the repo so there's something to check:

$ rm .git/hooks/pre-commit
$ git add .

Given how useful pre-commit is across projects I generally recommend installing via pip install --user, making it part of a tooling virtual environment, or using pipx:

$ pip install --user pre-commit
$ pipx install pre-commit

Next we'll need to create a YAML configuration file in the root of our repository called .pre-commit-config.yaml. To make things simple you can bootstrap a basic one via:

$ pre-commit sample-config > .pre-commit-config.yaml

As is the file looks something like this:

# See https://pre-commit.com for more information
# See https://pre-commit.com/hooks.html for more hooks
repos:
-   repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v3.2.0
    hooks:
    -   id: trailing-whitespace
    -   id: end-of-file-fixer
    -   id: check-yaml
    -   id: check-added-large-files

This already has some great entries for basic git related checks. Now we'll add a new entry to cover tox checking:

# See https://pre-commit.com for more information
# See https://pre-commit.com/hooks.html for more hooks
repos:
-   repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v3.2.0
    hooks:
    -   id: trailing-whitespace
    -   id: end-of-file-fixer
    -   id: check-yaml
    -   id: check-added-large-files
-   repo: local
    hooks:
    -   id: tox check
        name: tox-validation
        entry: pdm run tox
        language: system
        types: [python]
        pass_filenames: false

repo: local tells pre-commit that what I'm doing is not a managed action by code someone else wrote. It's completely custom code written by me. The code itself will run pdm run tox in the system environment when it finds python files have been modified. This is one of the more powerful features of pre-commit over the standard git hook. There's more flexibility on when certain commands get run. You don't need to check tests if only the README file was updated. Now to actually have this integrated with our workflow we'll need to run:

$ pre-commit install
pre-commit installed at .git/hooks/pre-commit

The generated file is simply an entry point to the actual pre-commit program which handles the processing of what we want to do:

#!/usr/bin/env bash
# File generated by pre-commit: https://pre-commit.com
# ID: 138fd403232d2ddd5efb44317e38bf03

# start templated
INSTALL_PYTHON=/home/johndoe/.pyenv/versions/py311/bin/python3.11
ARGS=(hook-impl --config=.pre-commit-config.yaml --hook-type=pre-commit)
# end templated

HERE="$(cd "$(dirname "$0")" && pwd)"
ARGS+=(--hook-dir "$HERE" -- "$@")

if [ -x "$INSTALL_PYTHON" ]; then
    exec "$INSTALL_PYTHON" -mpre_commit "${ARGS[@]}"
elif command -v pre-commit > /dev/null; then
    exec pre-commit "${ARGS[@]}"
else
    echo '`pre-commit` not found.  Did you forget to activate your virtualenv?' 1>&2
    exit 1
fi

Now after this I'll add the pre-commit YAML as there's a sanity check to ensure it's part of the repository:

$ git add .pre-commit-config.yaml

This is because the end goal is that anyone who downloads the code can install your pre-commit setup themselves and be able to run the appropriate tests. Now looking at the result:

$ git commit
[INFO] Installing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
Trim Trailing Whitespace.................................................Failed
- hook id: trailing-whitespace
- exit code: 1
- files were modified by this hook

Fixing README.md

Fix End of Files.........................................................Passed
Check Yaml...............................................................Passed
Check for added large files..............................................Passed
tox-validation...........................................................Failed

So as the file I purposely broke hasn't been fixed the tox-validation stage has failed. It also found an issue with trailing whitespace in my README.md and even fixed it for me. I'll go ahead and add the updated README and fix the commented out import so things are working again:

$ git add README.md
$ vim src/my_pdm_project_cwprogram_test/mymath.py
$ git add src/my_pdm_project_cwprogram_test/mymath.py
$ git commit
Trim Trailing Whitespace.................................................Passed
Fix End of Files.........................................................Passed
Check Yaml...............................................................Passed
Check for added large files..............................................Passed
tox-validation...........................................................Passed

Now that everything has passed I'm given the ability to commit. The output is also much more user friendly and mimics what you'd expect from a CI/CD or test suite run.

Working With Contributed pre-commit Code

In the configuration file you might have noticed:

repo: https://github.com/pre-commit/pre-commit-hooks

The basic functionality of this works similar to a GitHub repo with GitHub Actions code. You can even see the code for end_of_file_fixer as an example:

def fix_file(file_obj: IO[bytes]) -> int:
    # Test for newline at end of file
    # Empty files will throw IOError here
    try:
        file_obj.seek(-1, os.SEEK_END)
    except OSError:
        return 0
    last_character = file_obj.read(1)
    # last_character will be '' for an empty file
    if last_character not in {b'\n', b'\r'} and last_character != b'':
        # Needs this seek for windows, otherwise IOError
        file_obj.seek(0, os.SEEK_END)
        file_obj.write(b'\n')
        return 1

In fact there's actually a fairly sizeable amount of these listed on the pre-commit website. In fact another one is provided by the PDM project to make sure pdm.lock is up to date. I'll go ahead and add this in:

# See https://pre-commit.com for more information
# See https://pre-commit.com/hooks.html for more hooks
repos:
-   repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v3.2.0
    hooks:
    -   id: trailing-whitespace
    -   id: end-of-file-fixer
    -   id: check-yaml
    -   id: check-added-large-files
-   repo: local
    hooks:
    -   id: tox check
        name: tox-validation
        entry: pdm run tox
        language: system
        types: [python]
        pass_filenames: false
-   repo: https://github.com/pdm-project/pdm
    rev: 2.10.4 # a PDM release exposing the hook
    hooks:
    -   id: pdm-lock-check

Now I'll manually edit one of my dependencies in pyproject.toml and check the result:

dependencies = [
    "numpy>=1.25.1",
    "requests>=2.31.0",
]

$ git add pyproject.tmol .pre-commit-config.yaml
$ git commit
[INFO] Initializing environment for https://github.com/pdm-project/pdm.
[INFO] Installing environment for https://github.com/pdm-project/pdm.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
Trim Trailing Whitespace.................................................Passed
Fix End of Files.........................................................Passed
Check Yaml...............................................................Passed
Check for added large files..............................................Passed
tox-validation...........................................................Passed
pdm-lock-check...........................................................Failed
- hook id: pdm-lock-check
- exit code: 1

Lock file hash doesn't match pyproject.toml, packages may be outdated

In this case it noticed that my numpy dependency has changed. I'll go ahead and revert this and see what happens:

$ vim pyproject.toml
$ git add pyproject.toml
$ git commit
Trim Trailing Whitespace.................................................Passed
Fix End of Files.........................................................Passed
Check Yaml...............................................................Passed
Check for added large files..............................................Passed
tox-validation...........................................................Passed
pdm-lock-check...........................................................Passed

Now everything is looking good and I'm allowed to commit again. Trying to check if the pdm lock file was synced would have taken a substantial amount of time via a normal git hook, primarily due to not having context on the pdm code base. Instead I can use this hook provided by the developers in less than ten minutes to do it instead.

File Filtering

To showcase this best we'll want to commit what we have now so there's better control over what files get added:

$ git commit -m "Initial Commit"

Now since tox can run specific stages we can use this to break out tasks based on what's actually been committed. Considering when we'd want things to run:

All files should have the basic large files, trailing whitespace, and end of files check
README.md should have a markdown linter run against it
pyproject.toml should have a toml linter run against it
Linting should be run if files in src or tests are modified
Tests should be run if files in src or tests are modified
Documentation building should be run if files in src (for automodule generation) or docs are modified
Everything should run if pyproject.toml is updated as it controls settings and manages dependencies

So let's see what a solution like this would look like:

# See https://pre-commit.com for more information
# See https://pre-commit.com/hooks.html for more hooks
repos:
-   repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v3.2.0
    hooks:
    -   id: trailing-whitespace
    -   id: end-of-file-fixer
    -   id: check-yaml
    -   id: check-toml
    -   id: check-added-large-files
-   repo: local
    hooks:
    -   id: tox lint
        name: tox-validation
        entry: pdm run tox -e test,lint
        language: system
        files: ^src\/.+py$|pyproject.toml|^tests\/.+py$
        types_or: [python, toml]
        pass_filenames: false
    -   id: tox docs
        name: tox-docs
        language: system
        entry: pdm run tox -e docs
        types_or: [python, rst, toml]
        files: ^src\/.+py$|pyproject.toml|^docs\/
        pass_filenames: false
-   repo: https://github.com/pdm-project/pdm
    rev: 2.10.4 # a PDM release exposing the hook
    hooks:
    -   id: pdm-lock-check
-   repo: https://github.com/jumanjihouse/pre-commit-hooks
    rev: 3.0.0
    hooks:
    -   id: markdownlint

First off we have check-toml for linting our pyproject.toml file. markdownlint is taken from jumanjihouse/pre-commit-hooks. Note that check-* and the markdown linter generally has code to check if appropriate files were added so there's no need to add filters. Another thing to keep in mind is that it's recommended to pin rev against a specific revision (usually a tag) versus something like master or main. This will reduce "unexpected" surprises due to code changes. Here we see the actual filtering at work:

    -   id: tox lint
        name: tox-validation
        entry: pdm run tox -e test,lint
        language: system
        files: ^src\/.+py$|pyproject.toml|^tests\/.+py$
        types_or: [python, toml]

So files: ^src\/.+py$|pyproject.toml|^tests\/.+py$ is a regular expression to show what files we're interested in. In this case it's files under src/ and tests/ as well as pyproject.toml. types_or (requires 2.9.0 or later pre-commit) also ensures we're only looking at python or toml files. If you're wondering what to put in for types_or the identify-cli tool will let you know appropriate values that can be used:

$ identify-cli docs/source/index.rst
["file", "non-executable", "rst", "text"]

file is the most generic type. You can also be more specific with rst for example. types can be used if you want something that works off AND comparison instead:

        types: [python, executable]

This will run if a file is a python file and executable as well. Now that we've seen how comparisons work, it's time to put this into practice. I'll add our updated pre-commit YAML, modify README.md, and then run git commit:

$ vim README.md #changes here
$ git add .pre-commit-config.yaml README.md
$ git commit
[INFO] Initializing environment for https://github.com/jumanjihouse/pre-commit-hooks.
[INFO] Installing environment for https://github.com/jumanjihouse/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
Trim Trailing Whitespace.................................................Passed
Fix End of Files.........................................................Passed
Check Yaml...............................................................Passed
Check Toml...........................................(no files to check)Skipped
Check for added large files..............................................Passed
tox-validation.......................................(no files to check)Skipped
tox-docs.............................................(no files to check)Skipped
pdm-lock-check.......................................(no files to check)Skipped
Check markdown files.....................................................Passed

Given that no toml or appropriate python files were modified unnecessary checks are skipped according to our setup. The README.md file does have markdownlint run against it and has passed the checks. I'll go ahead and commit the file with the message "Updated README.md title". Now it's time to see what happens when we make changes to docs/:

$ vim docs/source/index.rst
$ git add docs/source/index.rst
$ git commit
Trim Trailing Whitespace.................................................Passed
Fix End of Files.........................................................Passed
Check Yaml...............................................................Passed
Check Toml...........................................(no files to check)Skipped
Check for added large files..............................................Passed
tox-validation.......................................(no files to check)Skipped
tox-docs.................................................................Passed

In this case tox-docs is run but since no python files are present neither linting nor tests were run. Now I'll revert the docs change and modify one of the tests. This should kick off the linting/tests phase but not do anything with the docs building:

$ git reset docs/source/index.rst
$ git checkout docs/source/index.rst
$ vim tests/test_mymath.py
$ git add tests/test_mymath.py
$ git commit
Trim Trailing Whitespace.................................................Passed
Fix End of Files.........................................................Passed
Check Yaml...............................................................Passed
Check Toml...........................................(no files to check)Skipped
Check for added large files..............................................Passed
tox-validation...........................................................Passed
tox-docs.............................................(no files to check)Skipped
pdm-lock-check.......................................(no files to check)Skipped
Check markdown files.................................(no files to check)Skipped

Finally we'll make an update to pyproject.toml which should run everything (I'll also reset the state of the test file so there are no false positives):

$ git reset tests/test_mymath.py
$ git checkout tests/test_mymath.py
$ vim pyproject.toml
$ git add .pre-commit-config.yaml pyproject.toml
$ git commit
Trim Trailing Whitespace.................................................Passed
Fix End of Files.........................................................Passed
Check Yaml...............................................................Passed
Check Toml...............................................................Passed
Check for added large files..............................................Passed
tox-validation...........................................................Passed
tox-docs.................................................................Passed
pdm-lock-check...........................................................Passed
Check markdown files.................................(no files to check)Skipped

Everything has been run. I will say that technically it wasn't necessary to run this as I only made a description update and nothing was changed in the actual dependencies or tool configuration. That means nothing was changed that would require linting/tests/documentation updates. Even so, trivial pyproject.toml changes will generally be very rare after the project is flushed out and you should only really be updating for dependency or tool configuration changes from there.

Conclusion

pre-commit is definitely one of the tools I wish I knew about sooner in my development career. It's a great way to avoid having people frown at your PRs for having 3 different lint fix only commits. Given that it is another roadblock to pushing commits you'll want to make sure your linting passes on the entire project via pre-commit run -a. Otherwise your coworkers will most definitely find a way to bypass it.

If you like what you see I'm am also available for hire. Those interested can find out more info in my dev.to profile.

Publishing Python Packages To PyPI With Twine

Chris White — Wed, 06 Dec 2023 21:21:06 +0000

PyPI Background
PyPI Account Creation
Package Refactor
Twine Upload
Adding Metadata
- Classifiers
Conclusion

In the last installment we looked at using tox to centralize our development tool workflow. We now have tests, n centralized way to check our code for issues, and documentation generation all in a single command. Now that our project is solid enough we need to look into releasing it to the general public. In this article we'll look at twine and using it to upload to the official PyPI servers.

PyPI Background

The first step to uploading code is having someplace to upload it to. Python's official package repository is PyPI. This is similar to NPM if you're familiar with NodeJS. It's a centralized location for both uploading and downloading python packages. Packages here are meant to handle everything from common to complex tasks so you don't have to write it yourself. It does mean, however, that you have to be careful about what you install. Malicious packages has been a known issue to not just PyPI, but many other popular package repositories as well. Packages without tests to show confidence that it works or documentation to show how you use it may cost you more time than you save. Finally, you don't want to be relying on code you expect to evolve over time if it's not longer being maintained.

PyPI Account Creation

Before we upload anything we'll need a PyPI account. In this case I actually recommend doing so on their test instance. This allows you to test your package uploads and installation without causing impact to the main PyPI servers. The sign in form is simple with the usual name, address, username, password, and captcha verification:

After completing the signup take note of the username as it will be required later. There's also an email verification step you'll need to complete to ensure you actually own the email address. After this you'll want to setup two factor authentication or 2FA for short. This means you login with two "factors": something you remember and something you own. The PyPI help page has information on setting this up for one of two methods:

Software based, such as Google Authenticator
Hardware based, such as a YubiKey

While YubiKeys are a bit on the pricey side the ability to authenticate by simply plugging in a device (and maybe taping a part of it) is very easy to do. Software solutions that work off of a smart phone can become an issue if your smart phone is misplaced or stolen. Once this is setup you'll want to go to your manage account page and there is an API token option towards the bottom:

Now tokens are the preferred way to authenticate against the system as if they are compromised you can simply generate a new one. If you use your account username and password instead and it's compromised getting your account back can be tedious. Clicking on Add API Token will lead to a screen that you can fill out like this:

On the test instance with your new account "Scope" only has one option so the warning can be ignored. Be sure to copy down the value as it will disappear afterwards and you'll have to regenerate it later if you forget. Now that our authentication details are available it's time to setup the upload process.

Package Refactor

Now up until now we've been using the name my_pdm_project to represent our package. Unfortunately the issue with this method is that PyPI allows one unique package name globally. This means if someone were to upload my_pdm_project you wouldn't be able to upload your package code as-is to PyPI. To work around this we're going to rename the project to my_pdm_project_[pypiusername] where [pypiusername] is replaced with the username you chose during the PyPI registration process. In this example I'll use my_pdm_project_cwprogram_test (be sure to not actually use this but the version with your username instead). Please note that this is only this one specific case for general tutorial usage and you normally wouldn't put your username in the package name like that. So let's go ahead and make the changes. The first thing you'll need to do is go to the src directory and rename your my_pdm_project to the new one:



$ cd src
$ mv my_pdm_project my_pdm_project_cwprogram_test

pyproject.toml needs to be updated to reflect the new package name. Also since we're here we're going to update the version to 1.0.0 as this is considered our finalized public release:



[project]
name = "my-pdm-project-cwprogram-test"
version = "1.0.0"

Test suites need to be updated to point to the new package location in tests/test_mymath.py:



from my_pdm_project_cwprogram_test.mymath import (
    add_numbers,
    average_numbers,

tox.ini needs an update so that code coverage is using the proper module name:



[testenv:test]
groups = testing
commands =
  pytest --cov=my_pdm_project_cwprogram_test

Sphinx will need updates in package name and version via docs/source/config.py:



project = 'my-pdm-project-cwprogram-test'
copyright = '2023, Chris White'
author = 'Chris White'

version = '1.0.0'
release = '1.0'

Finally we'll update the module name for our automatic api documentation in docs/source/mymath.rst:



Mymath Module Documentation
===========================
.. automodule:: my_pdm_project_cwprogram_test.mymath
    :members:

To make sure we didn't miss anything:



> pdm run tox
<snip>
  lint: OK (4.59=setup[0.84]+cmd[0.56,3.19] seconds)
  test: OK (3.59=setup[3.05]+cmd[0.55] seconds)
  docs: OK (3.55=setup[2.86]+cmd[0.69] seconds)
  congratulations :) (11.81 seconds)

Now that everything looks good it's time to begin the upload process (I also hope this is a good lesson on why you should stick with a project name once you've decided on it).

Twine Upload

Up until now I've recommended adding tools to our dev group in the PDM project. However twine is something where you want the authentication centralized and usable by all python environments. This fits the description of something that should be installed with pipx instead:

> pipx install twine

Now we need to configure authentication for twine to upload packages. For simplicity's sake I will be showing how to do this via a simple text file. Once you have gained more experience as a developer I highly recommend switching to keyring as a more secure method (the sooner the better) so your password isn't in a plain text file (I avoided doing it here as keyring can be overwhelming for new developers). To get started you'll need to create a .pypirc file in your home directory. For Linux and OSX you should be able to shortcut it as ~/.pypirc for Windows it will be in %HOME%\.pypirc in command prompt and $env:HOME\.pypirc in Powershell. Here is how the contents should look:



[distutils]
index-servers =
    testpypi

[testpypi]
repository = https://test.pypi.org/legacy/
username = __token__
password = pypi-[rest_of_token]

__token__ as the username is a special indicator that we're using an API token instead of a regular username and password. password is the API token value you obtained when generating it, which starts with pypi-. Now that our authentication is in place it's time to test uploading to PyPI. Note that unlike the other tools twine should only be run when you actually intend to do a release. This means I don't recommend putting it in tox.ini and instead running it manually. When you become more experienced this process will end up as part of a Continuous Integration / Continuous Deployment (CI/CD) automation pipeline. So now it's time to upload our package:



> pdm build
> twine upload -r testpypi dist/*

First we're building a package to upload which goes into the dist/ directory. Then we tell twine to upload the files in that directory. The -r testpypi tells twine to use the testpypi section we added to .pypirc earlier so it knows where to upload to. This is important as there are actually several potential candidates for uploading python packages to (private repositories such as AWS CodeArtifact and JFrog Artifactory for example). After the upload is complete you will see something like:



View at:
https://test.pypi.org/project/my-pdm-project-cwprogram-test/1.0.0/

Going to this page will show you the project page and how to install the package. I'll make a virtual environment real quick to show the installation:



> python3.11 -m venv pip_test_install_pypi
> pip_test_install_pypi/Scripts/activate
> pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ my-pdm-project-cwprogram-test 
Looking in indexes: https://test.pypi.org/simple/, https://pypi.org/simple/
Collecting my-pdm-project-cwprogram-test
  Obtaining dependency information for my-pdm-project-cwprogram-test from https://test-files.pythonhosted.org/packages/de/82/cce488c69f576b1ba270d5efc91381010f4a0bbbdf17fb6837c81adeb47b/my_pdm_project_cwprogram_test-1.0.0-py3-none-any.whl.metadata
  Using cached https://test-files.pythonhosted.org/packages/de/82/cce488c69f576b1ba270d5efc91381010f4a0bbbdf17fb6837c81adeb47b/my_pdm_project_cwprogram_test-1.0.0-py3-none-any.whl.metadata (274 bytes)
Collecting numpy>=1.25.2 (from my-pdm-project-cwprogram-test)
  Obtaining dependency information for numpy>=1.25.2 from https://files.pythonhosted.org/packages/da/3c/3ff05c2855eee52588f489a4e607e4a61699a0742aa03ccf641c77f9eb0a/numpy-1.26.2-cp311-cp311-win_amd64.whl.metadata
<snip>
Successfully installed certifi-2023.11.17 charset-normalizer-3.3.2 idna-3.6 my-pdm-project-cwprogram-test-1.0.0 numpy-1.26.2 requests-2.31.0 urllib3-2.1.0

So this installation method differs from the one shown on the project page in that it adds the actual PyPI repository as another package source since we're overriding it. Without this the package wouldn't install as it wouldn't be able to find the proper version of numpy.

Adding Metadata

As it is right now the package page is pretty bland because we haven't given enough information to describe our package. So it's time to open up pyproject.toml to add in a few things:



[project]
name = "my-pdm-project-cwprogram-test"
version = "1.0.1"
description = "A tutorial package for building python projects with PDM"
keywords = ["tutorial", "pdm"]
authors = [
    {name = "Chris White", email = "me@nospam.com.com"},
]
dependencies = [
    "numpy>=1.25.2",
    "requests>=2.31.0",
]
requires-python = ">=3.11"
readme = "README.md"
license = {text = "MIT"}

[project.urls]
Homepage = "https://dev.to/cwprogram"

Also the version needs to be updated to 1.0.1 in docs/source/conf.py. So we've updated the version and description as basic information. There's also a new keywords field which is useful for when a user is searching for packages. [project.urls] section was added to give a URL for the project (in this case it's just my dev.to page as there's no real project page). Now there's one more piece of useful information that can be added: classifiers.

Classifiers

PyPI and other packaging formats have the concept of classifiers. There's an exhausting yet comprehensive list on the pypi site. You can use it to tell users about what python your using, environmental constraints, integration with popular packages, operating systems, etc. Let's take a look at some of them we'll use:

Development Status :: 5 - Production/Stable: This is considered a stable release with unit tests, linting, etc.
Operating System :: OS Independent: The package doesn't use graphical interfaces or anything special and as such can run on major operating systems
Intended Audience :: Developers: The intended audience is developers who are looking to create their own python packages
License :: OSI Approved :: MIT License: Technically the license field already covers this but we'll let people know it's an MIT license as well
Programming Language :: Python :: 3.11: This project supports Python 3.11
Topic :: Education :: Testing: This is for educational purposes as a test project

The categories are good enough for our purposes. Once you get an actual project going I encourage you to look into these classifiers along with the keywords to make your project more easily discoverable by end users. Now for our finished product:



[project]
name = "my-pdm-project-cwprogram-test"
version = "1.0.1"
description = "A tutorial package for building python projects with PDM"
keywords = ["tutorial", "pdm"]
authors = [
    {name = "Chris White", email = "me@nospam.com"},
]
dependencies = [
    "numpy>=1.25.2",
    "requests>=2.31.0",
]
requires-python = ">=3.11"
readme = "README.md"
license = {text = "MIT"}
classifiers = [
  "Development Status :: 5 - Production/Stable",
  "Operating System :: OS Independent",
  "Intended Audience :: Developers",
  "License :: OSI Approved :: MIT License",
  "Programming Language :: Python :: 3.11",
  "Topic :: Education :: Testing"
]

[project.urls]
Homepage = "https://dev.to/cwprogram"

Finally re-uploading our new code:



> pdm build
> twine upload -r testpypi dist/*

Now if we look at the newer page you'll notice some differences:

So we have a reasonable description shown, our keywords are present, there's a homepage to point users to, and classifiers are available to give the user more metadata about our project. There are a few other fields and some ways to write fields differently, so I recommend looking over the official documentation on writing pyproject.toml for more information.

Conclusion

This ends the beginners python series, and it's been a fun ride. It took about two months of work and I hope it helps beginners out there get a start on setting up python projects in a fairly modern way. Once your more comfortable I recommend looking into:

GitHub to share your code
pre-commit to automate your tox runs with your git workflow
GitHub Actions for implementing CI/CD to automate your releases
readthedocs for hosting your sphinx documentation

Mastering these I would consider a step into being in an early intermediate level as a software developer. I hope you enjoyed this series and please look forward to more articles in the future! If you like what you see please check my dev.to profile as I'm open for work opportunities.

Automating Your Python Dev Tools With Tox

Chris White — Sun, 03 Dec 2023 22:34:09 +0000

Tooling Centralization
- Tox Configuration
- Tox Organization
- Tox Factors
Conclusion

So far the code for our beginner's project has become fairly stable and there are linting and testing tools to give confidence in basic functionality. It's almost at the level of being ready to release to the public. Before we do this, however, we need to fix one underlying issue to future project development: running all our tools manually. In this article we'll be looking at tox to centralize our tooling runs.

Tooling Centralization

So right now we have the following functionality as part of our python project:

flake8: generalized linting
pylint: more in depth linting
pytest: test suite and test coverage
sphinx: document generation

This is also the order we want to execute them as well. Running each of these manually every time isn't very efficient. To deal with this issue we'll be using the python tool tox. As tox is a development tool we'll need to install it as a traditional dev dependency with pdm. Along with that, there's actually a python package to make working with tox easier in a pdm project. I'll go ahead and install both:

> pdm add -dG dev tox tox-pdm

Tox Configuration

Now tox handles its configuration via an ini file called tox.ini. Let's go ahead and build this up slowly for each of the tools mentioned above. To start out create the tox.ini file in your project's toplevel directory. Then add the following:

[tox]
env_list = py311

This is declaring an environment that will be run under python 3.11. It's used on the backend for setting up a virtual environment. Next we'll setup the tests:

[tox]
env_list = py311

[testenv]
groups = dev
commands =
    flake8
    pylint --recursive=y .
    pytest --cov=my_pdm_project

We'll get to the docs in a later part as that requires an additional change. As far as the layout goes testenv is the default settings that tox uses for all environments. In this case the default is to install dev group dependencies and run the listed commands. env_list is something we'll look at in a bit but lets us diversify what we want to run. The current setting simply indicates that we wish to run the commands using python 3.11, shorthanded to py311.

Now before actually running anything tox will create a .tox folder when executed that acts much like our .venv folder. So this means we'll need to update flake8 and pylint to ignore it. Open up .flake8 in your project root directory and change the content to:

[flake8]
max-line-length = 99
exclude =
  .venv/*
  .tox/*
  docs/*

Then open up pyproject.toml to update this section to add the ignore as well:

[tool.pylint.MASTER]
ignore-paths = [ "^.venv/.*$", "^.tox/.*$", "^docs/*" ]

Note that I've also added docs/ because sphinx from our last article also uses python which we don't have much control over. Now for the moment of truth:

> pdm run tox
py311: commands[0]> flake8
py311: commands[1]> pylint --recursive=y .

--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)

py311: commands[2]> pytest --cov=my_pdm_project
<snip>
py311: OK (6.58=setup[3.38]+cmd[0.55,2.11,0.55] seconds)
congratulations :) (6.66 seconds)

I've focused the output but you can see tox is running all the normal commands for us. All of this is condensed into a single easy to run command!

Tox Organization

Now while everything is centralized, there's a few issues that can come up in larger projects:

Sometimes you just want to run only linting or only tests
Linting doesn't require the package to be installed, which tox will do every time

Fortunately for us there's a way to deal with it. tox supports the concept of multiple environments. That's why the first item is env_list. It also supports installation groups that we've defined in pyproject.toml (in this case the dev group with all of our dev tools). Up until now I've recommended installing everything that's not for the base code to be installed in the dev group for simplicity purposes. Now that we've come further in the project it's time to organize these for making our tox runs cleaner. Right now everything looks like this (save the versions of packages might be different):

[tool.pdm.dev-dependencies]
dev = [
    "flake8>=6.1.0",
    "pylint>=3.0.1",
    "pytest>=7.4.2",
    "pytest-cov>=4.1.0",
    "requests-mock>=1.11.0",
    "sphinx>=7.2.6",
    "tox>=4.11.4",
    "tox-pdm>=0.7.0",
]

tox and tox-pdm are fine as-is in the dev group. The rest we'll break up into linting, testing, and doc. Simply add each group as group_name = [] with the respective version entries inside. Let's do linting first:

[tool.pdm.dev-dependencies]
dev = [
    "pytest>=7.4.2",
    "pytest-cov>=4.1.0",
    "requests-mock>=1.11.0",
    "sphinx>=7.2.6",
    "tox>=4.11.4",
    "tox-pdm>=0.7.0",
]
lint = [
    "flake8>=6.1.0",
    "pylint>=3.0.1",
]

So here pylint and flake8 are now in a dedicate lint group. Now we'll do the same for testing and docs:

[tool.pdm.dev-dependencies]
dev = [
    "tox>=4.11.4",
    "tox-pdm>=0.7.0",
]
lint = [
    "flake8>=6.1.0",
    "pylint>=3.0.1",
]
testing = [
    "pytest>=7.4.2",
    "pytest-cov>=4.1.0",
    "requests-mock>=1.11.0",
]
docs = [
    "sphinx>=7.2.6",
]

As tox references the pdm.lock file, it will need to have all the groups refreshed that is listed within it. This can be done with:

> pdm lock -G:all

Now in tox.ini it's time to break everything up. We'll get everything we have currently updated first:

[tox]
env_list = lint, test

[testenv:lint]
groups = testing, lint
commands =
  flake8
  pylint --recursive=y .

[testenv:test]
groups = testing
commands =
  pytest --cov=my_pdm_project

The major difference here is that we have a testenv:lint and testenv:test. This is known as a "named environment". It's useful for cases of breaking out specific functionality. env_list will run in the order provided with lint first followed by test. For linting the reason why the testing packages is required is because pylint is running against our tests and as such needs to be able to resolve the modules we're using. Let's try this out with a quick run:

> pdm run tox
lint: install_deps> pdm sync --no-self --group testing --group lint
lint: commands[0]> flake8
lint: commands[1]> pylint --recursive=y .

--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)

lint: OK ✔ in 6.22 seconds
test: install_deps> pdm sync --no-self --group testing
test: commands[0]> pytest --cov=my_pdm_project
================================================= test session starts =================================================
tests\test_mymath.py .........                                                                                   [100%]

---------- coverage: platform win32, python 3.11.5-final-0 -----------
Name                                                     Stmts   Miss  Cover
----------------------------------------------------------------------------
.tox\test\Lib\site-packages\my_pdm_project\__init__.py       0      0   100%
.tox\test\Lib\site-packages\my_pdm_project\mymath.py        25      0   100%
----------------------------------------------------------------------------
TOTAL                                                       25      0   100%


================================================== 9 passed in 0.17s ==================================================
  lint: OK (6.22=setup[3.17]+cmd[0.55,2.50] seconds)
  test: OK (3.31=setup[2.75]+cmd[0.56] seconds)
  congratulations :) (9.61 seconds)

Again I've reduced the output to showcase the important parts. Since everything is broken up into separate parts, we can even choose to run only linting for example:

> pdm run tox -e lint
lint: install_deps> pdm sync --no-self --group testing --group lint
lint: commands[0]> flake8
lint: commands[1]> pylint --recursive=y .

--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)

  lint: OK (4.53=setup[0.86]+cmd[0.56,3.11] seconds)
  congratulations :) (4.62 seconds)

This is extremely useful for working on different parts of the development phase where you've passed linting but are trying to just get the tests to pass. Then once the tests are fixed you can run the whole suite to make sure everything as a whole looks fine. For the documentation generation it's slightly similar except we need to enter the docs directory first:

[tox]
env_list = lint, test, docs

[testenv:lint]
groups = testing, lint
commands =
  flake8
  pylint --recursive=y .

[testenv:test]
groups = testing
commands =
  pytest --cov=my_pdm_project

[testenv:docs]
groups = docs
changedir = docs
commands = sphinx-build source/ build/

Sphinx itself is installed thanks to be part of the docs group we setup in pyproject.toml. Running again we can see docs are now being generated:

docs: commands[0] C:\Users\johnsmith\my-pdm-project\docs> sphinx-build source/ build/
Running Sphinx v7.2.6
loading pickled environment... done
building [mo]: targets for 0 po files that are out of date
writing output...
building [html]: targets for 2 source files that are out of date
updating environment: 0 added, 0 changed, 0 removed
reading sources...
looking for now-outdated files... none found
preparing documents... done
copying assets... copying static files... done
copying extra files... done
done
writing output... [100%] mymath
generating indices... genindex py-modindex done
writing additional pages... search done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded.

The HTML pages are in build

Another issue here is that while sphinx and pytest need our package installed to work properly, linting is just working against the code itself. This means installation of our package just adds unnecessary time. We can skip the process by using skip_install = true in our lint section:

[testenv:lint]
groups = testing, lint
skip_install = true
commands =
  flake8
  pylint --recursive=y .

Running tox again shows it's no longer installing our package for the linting session:

> pdm run tox
lint: install_deps> pdm sync --no-self --group testing --group lint
lint: commands[0]> flake8
lint: commands[1]> pylint --recursive=y .

--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)

lint: OK ✔ in 4.59 seconds

Now unlike the previous definition where we had py311 there's nothing here indicating which version of python is used. In this case tox simply uses the one from the current environment (python 3.11 in this case, which is what we created the pdm virtual environment with). It turns out you can be explicit with python versions by utilizing an interesting feature called factors.

Tox Factors

This is somewhat of a fancy way of saying "names separated by hyphens", with a twist when it comes to python versions. I'm going to go ahead and make an isolated test case to showcase this:

[tox]
env_list = py311, py3.12

[testenv]
commands = python --version

Now to go ahead and run this:

> pdm run tox
py311: commands[0]> python --version
Python 3.11.5
py3.12: commands[0]> python --version
Python 3.12.0

So what's happening is that the format pyMajorMinor and pyMajor.Minor are special cases called "default factors" and they map to specific python versions. In this case py311 maps to python 3.11 and py3.12 maps to python 3.12. The tox documentation has the full details on special python interpreter factors. I will note that this only works if both python 3.11 and python 3.12 are installed on your system. Multiple python versions is an interesting topic, though might be somewhat more of an intermediate discussion. I've written about here if you're interested. This isn't just for python versions though, you can even get a bit more advanced by testing against multiple environment combinations. Let's say for example you made a webapp that you want to ensure works against database A and database B:

[tox]
env_list = py311-databaseA, py3.12-databaseB

[testenv]
commands = 
  python --version
  databaseA: python -c 'print("uses databaseA")'
  databaseB: python -c 'print("uses databaseB")'

So let's see what happens:

> pdm run tox
py311-databaseA: commands[0]> python --version
Python 3.11.5
py311-databaseA: commands[1]> python -c "print(\"uses databaseA\")"
uses databaseA
py311-databaseA: OK ✔ in 3.47 seconds

py3.12-databaseB: commands[0]> python --version
Python 3.12.0
py3.12-databaseB: commands[1]> python -c "print(\"uses databaseB\")"
uses databaseB

This is because what's actually happening is:

py311-databaseA = factor py311 + factor databaseA
py3.12-databaseB = factor py3.12 + factor databaseB

[testenv]
commands = 
  python --version
  databaseA: python -c 'print("uses databaseA")'
  databaseB: python -c 'print("uses databaseB")'

So here we can target specific factors, even if all are not included, to take on specific commands or install specific package dependencies. This makes tox an incredibly powerful tool. Even so our current needs are simple so leaving out the python version factors is fine as we're only testing the version set by pdm. Once you get farther in your python development career you'll be able to better understand how powerful the more advanced usages are.

Conclusion

My original intention was to combine this with uploading your package via twine. Then when I started writing it came to realization that the amount of explanation required would have made it a bit too verbose for my liking. I'd also like to note that I'm currently open for work if you like what you see. To not sound too spammy I'll just say check my dev.to profile for more information. In the next section we'll see the final installment of this series where we upload our python code for others to use.

Python Versions and Release Cycles

Chris White — Wed, 29 Nov 2023 15:49:11 +0000

Python 2 and 3
Python Version Components
Python Release Cycle
- End Of Life
- Security
- Bugfix
- Feature
Which Version To Use
- Beginners
- Companies
- Package Maintainers
Supporting Multiple Versions
Migrating Versions
Conclusion

As with all programming languages, the python has multiple versions and a release cycle to go with it. Understanding this release cycle is crucial as a python developer to reducing tech debt in your stack. This article will look at multiple python versions, release cycles, and making decisions on which version to utilize.

Python 2 and 3

If you've been in python long enough, you might have heard about the fun that is the python 2 to 3 transition. It was a fairly major revamp of the language which required migrations for a substantial portion of the userbase (myself included). It was so much that they ended up making a tool for it. This unfortunately left a sour taste for many developers as these kind of migrations can put pressure on developers employed in companies who are already attempting to meet deadlines. Needless to say it that many developers were not thrilled about it. After many extensions of timelines, python 2 received its end of life on January 1st 2020 with the python 2.7 series. The reason why I'm mentioning all of this is that it's definitely something python developers consider a not so great outcome and has had an impact on how they consider new version / large feature transitions.

Python Version Components

Python versions include a major, minor, and micro component. It's slightly similar in concept to semver, though the minor version may do something incompatible at times. An example of this is the removal of distutils in 3.12. While technically not a language change persay, it did mean some build systems had to be revamped to accommodate this change. In general you'll see python referred to in a major.minor format such as:

Python 2.7
Python 3.9
Python 3.12

New releases and security fixes are where you'll tend to find them referred to in full version format such as:

Python 2.7.17
Python 3.9.18
Python 3.12.0

Security releases use the full format so you know exactly which version(s) are affected. New releases show it so you can tell how many upgrade paths it would take to reach it. Since the release of python 3 in December of 2008 there hasn't been a new major release. This is due to requiring a substantial amount of incompatible changes. There hasn't been an plans announced for a python 4 at this time according to an interview with the language's creator. Though if there was it might be around reworking how the C API is handled which is a pain point for many.

Python Release Cycle

The python dev guide has a good breakdown of the release cycle process. Here is a screenshot of the current release cycle:

As shown here there are quite a number of end-of-life versions, followed by security, then bugfixes, and finally a feature version. All categories are referenced by their respective major.minor version. Let's take a look at the categories and what they entail.

End Of Life

Not much to say here, it basically means the version is not supported by python developers. No security, bugfixes, or feature additions will be added. Both source and binary version updates will no longer be available. You really want to avoid being on an end of life version, especially in corporate environment with security compliance programs.

Security

Security releases are where developers will only add security updates, so there will be no feature updates or non-security related bugfixes. They also won't have binaries available on the official website. You generally want to avoid being on the oldest security release as that means an end-of-life will be in the future.

Bugfix

These versions include fixes for bugs and security related issues. They also have official binaries available for them on the python website. If you're a new developer going for one of the bugfix versions would be ideal since binaries being available make them easier to install. The latest bugfix version would be ideal unless there's a chance that would require updates to popular packages. Bugfixes at some point will become security releases and lose their binary availability after the last bugfix release.

Feature

This is more of a bleeding edge release. It indicates new features planned for the future. As 3.12 was recently released as bugfix, 3.13 is now the feature release. One planned introduction in particular is an optional build mode that removes the global interpreter lock. It's generally recommended for beginners to avoid these since it's more of an in progress release where certain tutorials may be outdated.

Which Version To Use

So now the question becomes, which version should be utilized? Much of the answer lies in how it's utilized. One thing I will say is that no one should be using an end-of-life version (okay, maybe that one script from 2010 you wrote to parse movie titles). Let's take a look at some of the user categories and what version they might use.

Beginners

In general I recommend beginners be on the latest bugfix version. The exception to this is I wouldn't pick up a recent bugfix release if it hasn't been out for at least 4 months (3.12 at time of writing would fall under this). This is because tutorials might need updating, and some popular python packages might need time to migrate to it. It's better to play it safe. The main reason for recommending it is that binaries are widely available which make installing them very easy to pull off. This goes for both Windows and Mac in particular.

Companies

Recent startups may be closer to the bugfix versions as they don't have systems to migrate and being on newer versions is more advantageous tech debt wise. Enterprises generally stay within the security releases. Their focus is ensuring all their systems work as intended without unreasonable changes introduced. Sometimes there may be other limiting factors such as versions available through external solutions providers. AWS for example has a range of python versions for using with AWS lambda. They would want to be on at least 3.8 minimum given that there's a deprecation warning that went out recently for 3.7. There also might be requirements on what third party pip packages are available.

Package Maintainers

In general package maintainers will decide which versions to support based on what their userbase looks like and how many people are motivated/available to maintain said versions. At minimum package maintainers will generally use the oldest security release all the way up to the latest bugfix release (save python 3.12 which required some updates). Packages may require other packages which have their own version constraints and can also restrict which version is supported.

Supporting Multiple Versions

Package maintainers and other individuals may want to support multiple versions of python. For systems such as Windows and MacOSX you can download binaries of the bugfix versions. Security versions on the other hand don't have binaries officially available. In this particular case Linux is nice for working with multiple versions as packages are either installed from source or end binaries are build from source (especially important for security release where only the source is available). Distribution wise there are a few options:

Ubuntu has some bugfixes available along with a deadsnakes PPA. Being non-official the security implications should be considered, but it allows for easily installation of various python versions in a distro friendly way.
Fedora contains a wide range of python packages available through their built-in package manager. You could setup a quick dev box with Fedora on Raspberry Pi as one solution.

For OSX there is homebrew or pyenv (pyenv is another solution on Linux). As pyenv compiles from source it will require setting up XCode (the Apple IDE) tools to support this which can be pretty bulky. Windows users have chocolatey but the issue there is it works off the binaries. That means it won't have the latest security release available since those are source only. Conda is also another solution which can be picked up by Visual Studio Code as available versions of Python making development easier. In the end it might be best to consider using WSL on Windows for installing a Linux version and using that instead.

Migrating Versions

For companies I recommend looking into having upgrade path at regular intervals. The more you upgrade the less of a chance you'll be stuck with an EOL version. It also makes future upgrades easier, or at least gives you a buffer to handle issues. Having a CI/CD pipeline via something such as GitHun Actions along with a test suite makes this process even easier. Simply upgrade your pipeline to the new version then see what the test suite results are. If everything looks good you may be able to upgrade fairly easily. In the case of there being issues, you can at least have the gradual low priority tasks to investigate the issues and handle them at a better time. This is especially important in businesses with high priority deadlines.

Conclusion

This concludes a look into how to work with Python versions. It's something I was planning to go over in my beginners series until I realized it would be better flushed out on its own article. I will note that future articles will have a different direction to them as I'm currently in job hunt mode. This means that the direction will be more towards selling myself as a technically competent individual instead of the usual "I want to mess around with this interesting tech". This includes potentially doing AWS articles that I've tried to avoid here due to not being open source (the stack at least). If you're interested in hiring me for a remote full time position please look at my dev.to profile for more information.

Python Documentation With Docstrings and Sphinx

Chris White — Fri, 10 Nov 2023 17:30:52 +0000

pylint Prep
Docstrings
Documentation Generation With sphinx
Conclusion

In the last installment of the series we looked at how to achieve testing in our python projects. Much like testing, documentation can go a long way towards achieving adoption for your project. In this case we're going to be looking at how to generate documentation of your code using docstrings and sphinx.

pylint Prep

When doing the linting I explicitly disabled checking for docstrings. Now that we'll be using them, it's time to enable that check in the pyproject.toml file. First however, we'll want to check what the current state of our code is:

> pdm run pylint --recursive=y .
************* Module src.my_pdm_project.mymath
src\my_pdm_project\mymath.py:10:4: R1720: Unnecessary "elif" after "raise", remove the leading "el" from "elif" (no-else-raise)
src\my_pdm_project\mymath.py:16:10: W3101: Missing timeout argument for method 'requests.get' can cause your program to hang indefinitely (missing-timeout)
src\my_pdm_project\mymath.py:20:4: R1705: Unnecessary "else" after "return", remove the "else" and de-indent the code inside it (no-else-return)
************* Module tests.test_mymath
tests\test_mymath.py:23:11: C0123: Use isinstance() rather than type() for a typecheck. (unidiomatic-typecheck)
tests\test_mymath.py:24:11: C0123: Use isinstance() rather than type() for a typecheck. (unidiomatic-typecheck)

------------------------------------------------------------------
Your code has been rated at 9.25/10 (previous run: 9.25/10, +0.00)

So we see how the general development process works of writing code, checking with linting, and fixing things that come up. So first off is this section:

    if operation == '/' and b == 0:
        raise ZeroDivisionError
    elif operation not in SUPPORTED_OPERATIONS:
        raise ValueError

This is pretty simple, it just wants an if used instead of elif due to how exceptions handle breaks of logic flow. I'll go ahead and update it here:

    if operation == '/' and b == 0:
        raise ZeroDivisionError
    if operation not in SUPPORTED_OPERATIONS:
        raise ValueError

The next is that requests is being used without setting a timeout value. If the server was not responsive then our connection might end up in a stuck state. I'll go ahead and fix that here:

    res = requests.get(
        f'{BASE_URI}{operation_expression}', timeout=20
    )

This will throw an error if a response is not received within 20 seconds. Next is that the else here is redundant:

    if operation == '/':
        return float(res.text)
    else:
        return int(res.text)

I'll go ahead and update that here:

    if operation == '/':
        return float(res.text)
    return int(res.text)

Finally in our tests we're using type() instead of isinstance() which is cleaner. I'll go ahead and update the tests to do that:

    assert result == 5
    assert isinstance(result, int)
    assert isinstance(result2, float)

Now that everything is cleaned up I'll go ahead and run pylint again:

> pdm run pylint --recursive=y .

-------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 9.25/10, +0.75)

Now it's time to enable the docstring check. I'll go ahead and do this by removing the following portion from pyproject.toml:

[tool.pylint."MESSAGES CONTROL"]
disable = '''
missing-module-docstring,
missing-class-docstring,
missing-function-docstring
'''

Now I'll run pylint again:

> pdm run pylint --recursive=y .
************* Module src.my_pdm_project.mymath
src\my_pdm_project\mymath.py:1:0: C0114: Missing module docstring (missing-module-docstring)
src\my_pdm_project\mymath.py:9:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:25:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:29:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:33:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:37:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:41:0: C0116: Missing function or method docstring (missing-function-docstring)
************* Module tests.test_mymath
tests\test_mymath.py:1:0: C0114: Missing module docstring (missing-module-docstring)
tests\test_mymath.py:16:0: C0116: Missing function or method docstring (missing-function-docstring)
tests\test_mymath.py:27:0: C0116: Missing function or method docstring (missing-function-docstring)
tests\test_mymath.py:32:0: C0116: Missing function or method docstring (missing-function-docstring)
tests\test_mymath.py:39:0: C0116: Missing function or method docstring (missing-function-docstring)
tests\test_mymath.py:46:0: C0116: Missing function or method docstring (missing-function-docstring)
tests\test_mymath.py:52:0: C0116: Missing function or method docstring (missing-function-docstring)
tests\test_mymath.py:62:0: C0116: Missing function or method docstring (missing-function-docstring)

-------------------------------------------------------------------
Your code has been rated at 7.61/10 (previous run: 10.00/10, -2.39)

So as you can see there's a lot of output. Now let's talk about how to go about fixing this.

Docstrings

Docstrings are done by enclosing text in triple double quotes. As an example:

def add_numbers(a: int, b: int):
    """
    This is my function
    """
    return make_mathjs_request(a, b, '+')

Now the official docstring specification is part of PEP 257. While it does describe the overall format it's not specific about the format of what you would put in a doc string. In this case I'm going to be utilizing sphinx as the code documentation generator of choice. Now let's look at what our function's doc string will become using the sphinx format:

def add_numbers(a: int, b: int):
    """Add two numbers together

    :param a: The base integer to use in the add operation
    :type a: int
    :param b: The integer to add to the base integer
    :type b: int

    :return: The sum of both integers
    :rtype: int
    """

:param: indicates what a parameter is meant for and :type: indicates the type of the parameter. :return: will describe the return of the function and :rtype: the return value. Now after implementing this docstring and running pylint again:

> pdm run pylint --recursive=y .
************* Module src.my_pdm_project.mymath
src\my_pdm_project\mymath.py:1:0: C0114: Missing module docstring (missing-module-docstring)
src\my_pdm_project\mymath.py:9:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:39:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:43:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:47:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:51:0: C0116: Missing function or method docstring (missing-function-docstring)
************* Module tests.test_mymath
tests\test_mymath.py:1:0: C0114: Missing module docstring (missing-module-docstring)
tests\test_mymath.py:16:0: C0116: Missing function or method docstring (missing-function-docstring)
tests\test_mymath.py:27:0: C0116: Missing function or method docstring (missing-function-docstring)
tests\test_mymath.py:32:0: C0116: Missing function or method docstring (missing-function-docstring)
tests\test_mymath.py:39:0: C0116: Missing function or method docstring (missing-function-docstring)
tests\test_mymath.py:46:0: C0116: Missing function or method docstring (missing-function-docstring)
tests\test_mymath.py:52:0: C0116: Missing function or method docstring (missing-function-docstring)
tests\test_mymath.py:62:0: C0116: Missing function or method docstring (missing-function-docstring)

------------------------------------------------------------------
Your code has been rated at 7.76/10 (previous run: 7.61/10, +0.15)

We can see there is a slight increase in our overall score since we added the docstring. Now one issue here is that pylint is expecting the tests to have docstrings. Functionality wise we really don't need them in our tests since the point of code documentation is to show the users how the code they're consuming works. A general user isn't going to be consuming tests. At the top of the test file I can tell pylint to ignore docstrings for them since I still want it to make sure the other parts of my tests are solid:

# pylint: disable=missing-docstring
from urllib.parse import quote_plus
import pytest
import requests_mock

from my_pdm_project.mymath import (
    add_numbers,

Now after running pylint:

> pdm run pylint --recursive=y .
************* Module src.my_pdm_project.mymath
src\my_pdm_project\mymath.py:1:0: C0114: Missing module docstring (missing-module-docstring)
src\my_pdm_project\mymath.py:9:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:39:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:43:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:47:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:51:0: C0116: Missing function or method docstring (missing-function-docstring)

------------------------------------------------------------------
Your code has been rated at 9.05/10 (previous run: 8.96/10, +0.09)

So I'm a big closer here. Now besides the functions it's also mentioning a module docstring. At the top of our python code we can simply write a description of what the underlying code is meant to do:

"""
A module containing simple math operations.
"""
from urllib.parse import quote_plus
import numpy as np
import requests

Now another check:

> pdm run pylint --recursive=y .
************* Module my_pdm_project.mymath
src\my_pdm_project\mymath.py:12:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:42:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:46:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:50:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:54:0: C0116: Missing function or method docstring (missing-function-docstring)

------------------------------------------------------------------
Your code has been rated at 9.21/10 (previous run: 9.05/10, +0.16)

So now just the function docstrings are left. Let's look at a more complex example for make_mathjs_request:

def make_mathjs_request(a: int, b: int, operation: str):
    """Make a expression call against the MathJS API

    :param a: Base integer for the operation
    :type a: int
    :param b: Integer to use with a in the operation
    :type b: int
    :param operation: Operation to run against a and b
    :type operation: str

    :raises ZeroDivisionError: Raised if division by 0
    :raises ValueError: Raised if not a supported operation

    :returns:
        - int for non division operations
        - float for division operations
    """

Here it's somewhat like what we saw before. What's new now is that we also documentation exceptions that can be raised, and why they'd be raised. The :returns: is used since we're returning either a float or int depending on the operation. As mentioned before this was done as an example case for showing testing and normally you wouldn't want a function to return multiple types. After a pylint run:

> pdm run pylint --recursive=y .
************* Module my_pdm_project.mymath
src\my_pdm_project\mymath.py:58:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:62:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:66:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:70:0: C0116: Missing function or method docstring (missing-function-docstring)

------------------------------------------------------------------
Your code has been rated at 9.37/10 (previous run: 9.21/10, +0.16)

So I just need to add docstrings to the other functions. After all is done the file looks like this:

"""
A module containing simple math operations.
"""
from urllib.parse import quote_plus
import numpy as np
import requests

BASE_URI = "http://api.mathjs.org/v4/?expr="
SUPPORTED_OPERATIONS = ['+', '-', '*', '/']


def make_mathjs_request(a: int, b: int, operation: str):
    """Make a expression call against the MathJS API

    :param a: Base integer for the operation
    :type a: int
    :param b: Integer to use with a in the operation
    :type b: int
    :param operation: Operation to run against a and b
    :type operation: str

    :raises ZeroDivisionError: Raised if division by 0
    :raises ValueError: Raised if not a supported operation

    :returns:
        - int for non division operations
        - float for division operations
    """
    if operation == '/' and b == 0:
        raise ZeroDivisionError
    if operation not in SUPPORTED_OPERATIONS:
        raise ValueError

    operation_expression = quote_plus(f'{a}{operation}{b}')
    res = requests.get(
        f'{BASE_URI}{operation_expression}', timeout=20
    )

    if operation == '/':
        return float(res.text)
    return int(res.text)


def add_numbers(a: int, b: int):
    """Add two numbers together

    :param a: The base integer to use in the add operation
    :type a: int
    :param b: The integer to add to the base integer
    :type b: int

    :return: The sum of both integers
    :rtype: int
    """
    return make_mathjs_request(a, b, '+')


def subtract_numbers(a: int, b: int):
    """Subtract two numbers

    :param a: The base integer to use in the subtract operation
    :type a: int
    :param b: The integer to subtract the base integer from
    :type b: int

    :return: The subtraction of both numbers
    :rtype: int
    """
    return make_mathjs_request(a, b, '-')


def multiply_numbers(a: int, b: int):
    """Multiple two numbers together

    :param a: The base integer to use in the multiply operation
    :type a: int
    :param b: The integer to multiply against the base number
    :type b: int

    :return: The result of multiplying both numbers
    :rtype: int
    """
    return make_mathjs_request(a, b, '*')


def divide_numbers(a: int, b: int):
    """Divide two numbers

    :param a: The base integer to use in the add operation
    :type a: int
    :param b: The integer divide a by
    :type b: int

    :return: The quotient of the division operation
    :rtype: float
    """
    return make_mathjs_request(a, b, '/')


def average_numbers(numbers: list[int]):
    """Average a list of numbers

    :param numbers: The list of numbers to average
    :type numbers: list[int]

    :return: The average of the numbers
    :rtype: float
    """
    return np.average(numbers)

After finishing up documenting everything here we can check what pylint has to say:

> pdm run pylint --recursive=y .

-------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 9.37/10, +0.63)

Everything is in the clear now!

Documentation Generation With sphinx

Now as is the code documentation is a bit difficult to work with for an average user. To help with this we can use sphinx to take our docstrings and generate them into various formats. As with our other tools this will be added as a development package:

> pdm add -dG dev sphinx

Now we'll need to create a document directory for where our documentation will be stored.

> mkdir docs

Now it's time to setup sphinx to generate documentation. The thing to keep in mind with sphinx is it's primarily a documentation generation tool and generation of library documentation is a side bonus. With this in mind we'll go ahead and setup how our project will work vi the nice sphinx-quickstart utility:

> cd docs
> pdm run sphinx-quickstart --no-makefile -M --ext-autodoc -p "my-pdm-project" -a "Chris White" -v "0.3.0" -r "0.3.0" -l "en" --sep .

So there's a few things to digest here. --no-makefile and -M are done to avoid using make for building. This was mostly to avoid adding in another thing to install. -p sets the name of the project, -a the author, -v and -r are for version and release. They're the same right now because there's no 1.0 release yet, but if there was I'd recommend something like 1.0 for the version and 1.0.0 for the release. It's somewhat like how there's 3.12 for python but the 3.12 version has several releases under it. -l sets the language of the project and --sep ensures that source and build directories are separate. I tend to prefer this because it's easier to ignore the build directory through things like .gitignore later on. Finally . indicates the directory of the project, or more specifically the "documentation project" (as supposed to the code project where all our code is). Now that everything is setup we can use sphinx-build to generate html for us:

> pdm run sphinx-build source/ build/

Now if I look in the build directory there will be an index.html I can access via a browser:

Right now there isn't much going on and "Module Index" doesn't work because it hasn't been setup to recognize our docstrings. To do this we'll create a new file in docs/source/ called mymath.rst with the following content:

Mymath Module Documentation
===========================
.. automodule:: my_pdm_project.mymath
    :members:

Now this format is known as rst or reStructuredText. It's a format that's more feature rich than markdown and useful for structured documentation. In this case it's referring to a function in rst. These functions are in the form:

.. function_name:: arguments
    :option: value

    content

In the case of automodule it will generate documentation for my_pdm_project.mymath including all of its members. Now in the same directory there will be an index.rst file that needs to be edited like so:

.. my-pdm-project documentation master file, created by
   sphinx-quickstart on Fri Nov 10 09:01:15 2023.
   You can adapt this file completely to your liking, but it should at least
   contain the root `toctree` directive.

Welcome to my-pdm-project's documentation!
==========================================
.. toctree::
   :maxdepth: 2
   :caption: Contents:

   mymath


Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

What's changed is that mymath has now been added to our table of contents. Note that contents are indicated by a blank line after the options. In this case the content is mymath on a single line. sphinx knows that this is referencing mymath.rst and toctree will automatically parse mymath.rst to provide a table of contents. Now after running this again in the docs directory:

> pdm run sphinx-build source/ build/

The main index.html page will show:

And clicking on "Mymath Module Documentation" will show the documentation generated via our doc strings:

Conclusion

This concludes a look at documentation generation via sphinx parsing python docstrings. I will say that rst is more involved than simple markdown, but it's feature rich nature makes it ideal for many forms of documentation structure. In the next section we'll be looking at orchestrating all of our tools so far and uploading our code for everyone to use.

Beginners Python Series Updates and Writing Motivation

Chris White — Sun, 29 Oct 2023 13:09:25 +0000

The next installment of the beginner python series will be delayed due to needing to backtrack. My original though was using tox to automate testing with the various tools I've gone over. Then I realized midway that discussing code documentation would be a better next step after testing so I'm back to the drawing board again progress wise. This means there might not be another update for a week/week and a half. Once this is done there will be one more article before I think I'm done with the series.

Why I Write

I haven't really touched on what motivates me to write technical articles. Most of my technical writing started back in the days when I was part of the Gentoo Linux project. I showed how to setup certain packages and actually helped the Japanese team with translations. From there it just kind of stuck as "my thing" in terms of contributions. So much in fact that one of my previous jobs hired me based on my passion for documentation.

Now I did have a time where I thought about monetizing articles (also finding out how so companies tend to "forget" they're supposed to be paying you). As I've aged like a not so fine wine I started to contemplate how I really wanted to give back to the community in general. That's when I started writing docs on dev.to, for free, no strings attached. This article and many others are available on GitHub under creative commons licensing. Some of the more on key reasons I provide them for free:

Doing it for money causes pressure
If I attempted to monetize it, that might mean that the college students who are a sizeable majority of my readers might not have as much access to it
My content is more attached to me in a way that I have control over

Now, if I was about to go bankrupt levels of financial trouble then yes I might consider trying to make money off content. Until then my current job is enough to keep things running.

Writing For Beginners

Another side of this story is that I've been doing a lot of off the side hobby things as I've finally hit that point of "wow I might get burned out soon!". Part of it is that writing for beginners is more effort than writing to the more technically savvy users. For example, in the first installment of the beginning python series I showed how to install python for several different operating systems in a way that would be straightforward for a new user to get up and running. If I was working with more adept users I'd instead just point to the install guide and let them figure it out. I also am making sure that I test and run every command I talk about to make sure there are no surprises.

More Contributing

Lately I've also been wanting to do more open source work, but writing articles kind of takes away from that. This includes my PyPyInstaller project for windows as well as a python project for creating self signed certs programmatically. It also would be nice to contribute to various projects in terms of bug wrangling. This most likely means articles will go from a one week pace to "whenever my time is free".

Wrapping Up

So basically expect the pacing to be a lot slower than the weekly speed I've been managing so far. In fact I'm actually surprised I managed to come this far without terrible burn out. I'll see where the future takes me and hopefully some future articles will be around open source contributions.

Testing and Refactoring With pytest and pytest-cov

Chris White — Sun, 22 Oct 2023 12:03:30 +0000

Why Test?
pytest
- Setup
- Test Refactor
- Parameterize
Refactoring
- Coverage With pytest-cov
- Working With Mathjs
- Versions
- Working With New Tests
- Mocks
Conclusion

In the last installment we looked at how to utilize linting tools to improve overall quality. Now it's time to look at making sure our code is doing what it's supposed to be. In this article we'll look at adding unit testing to improve overall code, and even do some refactoring to see how our tests evolve. I will say that this article will be longer than most as I feel that understanding testing is that valuable.

Why Test?

Testing is something that seems more like a chore if you've been working solo for most of your programming lifetime. It's when you start to do things like contribute to open source or work in a team environment that the value of testing becomes more apparent. For both situations it's not uncommon for users to submit code to be reviewed. In team situations there can often be various programming styles which can create hurdles for code reviews. Having tests that can be run helps alleviate some of the concerns of reviewing code (not replace it mind you) and increase the chance that code gets approved sooner. It also gives a level of assurance when doing more risky changes such as code refactoring.

pytest

Up until now we've been using python's unittest module. This was chosen as a first step since it comes with python out of the box. Now that we've gone over dev dependencies I think it's a good time to look at pytest as a unit test alternative. I highly recommend getting accustomed to pytest as it's used quite often in the python ecosystem to handle testing for projects. It's also a bit more user friendly in how it discovers and runs tests.

Setup

To begin I'll add pytest as a dev dependency to our current project:

$ pdm add -dG dev pytest

Now one of the amazing features of pytest is that our tests built with unittest actually work right out of the box:

$ pdm run pytest
============================================================================================================= test session starts ==============================================================================================================
collected 5 items

tests/test_mymath.py .....                                                                                                                                                                                                                [100%]

============================================================================================================== 5 passed in 0.34s ===============================================================================================================

This feature allows us to gradually migrate tests made with the unittest over to how pytest expects things.

Test Refactor

Refactoring our previous unittest test code to pytest doesn't actually change too function wise:

import pytest

from my_pdm_project.mymath import (
    add_numbers,
    average_numbers,
    subtract_numbers,
    multiply_numbers,
    divide_numbers
)


def test_add():
    assert add_numbers(2, 3) == 5


def test_subtract():
    assert subtract_numbers(0, 3) == -3
    assert subtract_numbers(5, 3) == 2


def test_multiply():
    assert multiply_numbers(3, 0) == 0
    assert multiply_numbers(2, 3) == 6


def test_divide():
    assert divide_numbers(6, 3) == 2.0
    with pytest.raises(ZeroDivisionError):
        divide_numbers(3, 0)


def test_average():
    assert average_numbers([90, 88, 99, 100]) == 94.25

The main difference is using functions instead of a class and methods and pytest.raises to check for a division by zero exception.

Parameterize

One common aspect of testing is to test the same piece of code, but with different inputs. You can see that with one of our unit tests:

def test_subtract():
    assert subtract_numbers(0, 3) == -3
    assert subtract_numbers(5, 3) == 2

One interesting feature of pytest is to be able to centralize this form of code layout. As an example:

@pytest.mark.parametrize("x,y,expected", [(0, 3, -3), (5, 3, 2)])
def test_subtract(x, y, expected):
    assert subtract_numbers(x, y) == expected

So the way this works is pytest utilizes a python language feature called decorators. As the name implies they decorate functions and methods to enhance the underlying functionality. This can be used for features such as mapping a web service URL to a python function. In this case it's enhancing how a test runs. Now the first argument to this decorate is a list of argument names to map our declared values to. The second argument is the the values we intend to use as a list of tuples. So in essence the test gets run as:

First Run: x = 0, y = 3, expected = -3
Second Run: x = 5, y = 3, expected = 2

This easily allows for consolidating repetitive test portions. In fact you could even do that for the functions themselves, save the divide by zero test due to its special handling:

import pytest

from my_pdm_project.mymath import (
    add_numbers,
    average_numbers,
    subtract_numbers,
    multiply_numbers,
    divide_numbers
)


@pytest.mark.parametrize("method_to_test,x,y,expected", [
    (add_numbers, 2, 3, 5),
    (subtract_numbers, 0, 3, -3),
    (subtract_numbers, 5, 3, 2),
    (multiply_numbers, 3, 0, 0),
    (multiply_numbers, 2, 3, 6),
    (divide_numbers, 6, 3, 2.0)
])
def test_operations(method_to_test, x, y, expected):
    assert method_to_test(x, y) == expected


def test_divide_by_zero():
    with pytest.raises(ZeroDivisionError):
        divide_numbers(3, 0)


def test_average():
    assert average_numbers([90, 88, 99, 100]) == 94.25

With functions being objects in python, they can be passed in like other variables, then be called as if they were the functions referenced themselves. While this is a rather fancy solution It may cause difficulty for others trying to read your code. To make things simple I'll go back to the more verbose form which is easier to follow along:

import pytest

from my_pdm_project.mymath import (
    add_numbers,
    average_numbers,
    subtract_numbers,
    multiply_numbers,
    divide_numbers
)


def test_add():
    assert add_numbers(2, 3) == 5


@pytest.mark.parametrize('x,y,expected', [(0, 3, -3), (5, 3, 2)])
def test_subtract(x, y, expected):
    assert subtract_numbers(x, y) == expected


@pytest.mark.parametrize('x,y,expected', [(3, 0, 0), (2, 3, 6)])
def test_multiply(x, y, expected):
    assert multiply_numbers(x, y) == expected


def test_divide():
    assert divide_numbers(6, 3) == 2.0
    with pytest.raises(ZeroDivisionError):
        divide_numbers(3, 0)


def test_average():
    assert average_numbers([90, 88, 99, 100]) == 94.25

Refactoring

Now to highlight testing I'll go ahead and start doing a refactor. In this case I've found that there's a simple REST API called mathjs which provides math functionality. While not very practical functionality wise it does show the common concept of reaching out to a web API. Now before starting with the refactor, it's time to consider another interesting point.

Coverage With pytest-cov

Coverage is the concept of comparing the code you wrote to the tests you wrote. It helps answer the question "did you really test everything?". pytest thankfully has a plugin called pytest-cov which will add this functionality to pytest that we use already. Like other tools this is a dev dependency:

$ pdm add -dG dev pytest-cov

A quick run will show how we look right now:

$ pdm run pytest --cov=my_pdm_project
================================================= test session starts =================================================
plugins: cov-4.1.0
collected 7 items

tests/test_mymath.py .......                                                                                     [100%]

---------- coverage: python 3.11.5-final-0 -----------
Name                             Stmts   Miss  Cover
----------------------------------------------------
src/my_pdm_project/__init__.py       0      0   100%
src/my_pdm_project/mymath.py        11      0   100%
----------------------------------------------------
TOTAL                               11      0   100%

According to the report I have 100% coverage, meaning I have enough tests for the code that's been written. The --cov=my_pdm_project argument enables code coverage based on anything under the module my_pdm_project. Now that we have a way to measure that we've covered everything with our tests, it's time to refactor.

Working With Mathjs

Since this is a web API we'll need to setup an HTTP client. requests is considered the go to package for making web requests in python. It's simple to use and very feature rich. Since it will now be required for the application to work, we'll go ahead and add it to our project:

$ pdm add requests

So the first thing that will need to happen is understanding how to send data. Looking at the website it states that an expr query parameter needs to be sent with a value of the expression we want to evaluate URL encoded. This URL encoding is a way to escape characters so they aren't misinterpreted for parts of the URL itself. This can be achieved in python through urllib.parse.quote_plus. So putting things together we have:

BASE_URI = "http://api.mathjs.org/v4/?expr="


def make_mathjs_request(expression: str):
    res = requests.get(
        f'{BASE_URI}{quote_plus(expression)}'
    )
    return int(res.text)

So this takes an expression as a string, then passes it to the API as a URL encoded value. The f before the string declares it as a formatted string literal or f-string. This form of a string allows you to insert expression into a string which will be filled with the actual value. It's easier to read than the old format method which put temporary placeholders that would be defined at the end of the string. Finally, it will convert it to an integer value since it makes sense that a math library would return, well, numbers. Right now our code will looks like this:

from urllib.parse import quote_plus
import numpy as np
import requests

BASE_URI = "http://api.mathjs.org/v4/?expr="


def make_mathjs_request(expression: str):
    res = requests.get(
        f'{BASE_URI}{quote_plus(expression)}'
    )
    return int(res.text)


def add_numbers(a: int, b: int):
    return a + b


def subtract_numbers(a: int, b: int):
    return a - b


def multiply_numbers(a: int, b: int):
    return a * b


def divide_numbers(a: int, b: int):
    return a / b


def average_numbers(numbers: list[int]):
    return np.average(numbers)

Versions

Now right now we've made a pretty decent change to the code. Because of this we'll also want to increase our version number of the package. How to handle versions of software can be a very opinionated story. Part of this reason is that versions are a way to understand how your development lifecycle works. For example versions may tell you that a project:

Values stability, having a new version in slower time periods, even quarterly
Values bleeding edge development where users may want new features sooner and don't want to wait
Doesn't have a public release yet and is still in the core development phase
Whatever OpenSSL is doing

Now looking at our current pyproject.toml the version is declared as version = "0.1.0". I tend to refer to this as a "pre-release" version where you're not ready for general public consumption of the end result yet. Once a release is made for general consumption (1.0.0) is when many developers abide by a semver like system of version numbers. For right now we're not quite ready to push this to the public yet so I'll go ahead and increment to 0.2.0:

[project]
name = "my-pdm-project"
version = "0.2.0"
description = ""

Now I can install the newer version to ensure everything works properly:

$ pdm install

Working With New Tests

So now that we've handled versioning, it's time to check what our coverage looks like:

$ pdm run pytest --cov=my_pdm_project
================================================= test session starts =================================================
plugins: cov-4.1.0
collected 7 items

tests/test_mymath.py .......                                                                                     [100%]

---------- coverage: platform win32, python 3.11.5-final-0 -----------
Name                             Stmts   Miss  Cover
----------------------------------------------------
src/my_pdm_project/__init__.py       0      0   100%
src/my_pdm_project/mymath.py        15      2    87%
----------------------------------------------------
TOTAL                               15      2    87%

Now it's says there are two misses. This is because we've added new code to our module without a test. If we run it again with --cov-report term-missing added as an argument it will tell us what exactly was missed:

Name                             Stmts   Miss  Cover   Missing
--------------------------------------------------------------
src\my_pdm_project\__init__.py       0      0   100%
src\my_pdm_project\mymath.py        17      2    88%   9-12
--------------------------------------------------------------
TOTAL                               16      2    88%

So in this case lines 9-12 of src/my_pdm_project/mymath.py is the issue. The lines in question are from the new MathJS client:

res = requests.get(
    f'{BASE_URI}{quote_plus(expression)}'
)
return int(res.text)

Now up until now tests were mostly copy paste without much afterthought. This time though I'll show how the process generally works. First we'll consider two things we want to be assured of:

An expression given to the API gives a proper response
The type is int

So here's an example of what that looks like:

def test_makemathjs_request():
    result = make_mathjs_request("2+3")
    assert result == 5
    assert type(result) is int

Now when dealing with first time tests the recommendation is you cause your test to fail on purpose. This is to help avoid situations where you're running against a flawed test. I'll go ahead and change result to check against 6 which should fail:

def test_makemathjs_request():
    result = make_mathjs_request("2+3")
    assert result == 6
    assert type(result) is int

Now a quick run to validate:

tests/test_mymath.py F.......                                                                                                                                                      [100%]

======================================================================================= FAILURES ========================================================================================
________________________________________________________________________________ test_makemathjs_request ________________________________________________________________________________

    def test_makemathjs_request():
        result = make_mathjs_request("2+3")
>       assert result == 6
E       assert 5 == 6

tests/test_mymath.py:15: AssertionError

As intended our test comes back and lets us know that the result of 5 is not in fact the value 6 we're checking against. Now it's time to use the working test instead since we know we're not testing against flawed logic:

def test_makemathjs_request():
    result = make_mathjs_request("2+3")
    assert result == 5
    assert type(result) is int

pytest shows everything is good and there are no code coverage issues either:

$ pdm run pytest --cov=my_pdm_project  --cov-report term-missing
================================================================================== test session starts ==================================================================================
plugins: cov-4.1.0
collected 8 items

tests/test_mymath.py ........                                                                                                                                                      [100%]

---------- coverage: platform win32, python 3.11.5-final-0 -----------
Name                             Stmts   Miss  Cover   Missing
--------------------------------------------------------------
src/my_pdm_project/__init__.py       0      0   100%
src/my_pdm_project/mymath.py        16      0   100%
--------------------------------------------------------------
TOTAL                               16      0   100%

Mocks

Now one issue with the code right now is that it's using live data from MathJS' servers. Given that we're going to be porting the other math functions to use MathJS that means that our tests will hit the servers several times. This has a chance to be flagged and our connection throttled. Such a situation would cause an unintended failure in our tests. Thankfully in testing there's the ability to fake a live service, known as mocking. In this case there is a python package available called requests-mock which will help us achieve this goal. As with other test related packages we'll install it as a dev dependency:

$ pdm add -dG dev requests-mock

Now the way the mocking works is we pass in what URL we're expecting to be called and requests-mock will capture it and return mock data instead. Let's take a look at what that entails:

from urllib.parse import quote_plus
import pytest
import requests_mock

from my_pdm_project.mymath import (
    add_numbers,
    average_numbers,
    subtract_numbers,
    multiply_numbers,
    divide_numbers,
    make_mathjs_request,
    BASE_URI
)


def test_makemathjs_request():
    with requests_mock.Mocker() as m:
        m.get(f'{BASE_URI}{quote_plus("2+3")}', text='5')
        result = make_mathjs_request("2+3")
    assert result == 5
    assert type(result) is int

so as shown here we're capturing the URL that should be called when make_mathjs_request is passed "2+3" and return "5" as test data. Then as before we check if the result is 5 and the type returned is an integer. Now that our request related code is solid it's time to create a complete package and test suite where everything uses MathJS:

from urllib.parse import quote_plus
import numpy as np
import requests

BASE_URI = "http://api.mathjs.org/v4/?expr="


def make_mathjs_request(expression: str):
    res = requests.get(
        f'{BASE_URI}{quote_plus(expression)}'
    )
    return int(res.text)


def add_numbers(a: int, b: int):
    return make_mathjs_request(f'{a}+{b}')


def subtract_numbers(a: int, b: int):
    return make_mathjs_request(f'{a}-{b}')


def multiply_numbers(a: int, b: int):
    return make_mathjs_request(f'{a}*{b}')


def divide_numbers(a: int, b: int):
    return make_mathjs_request(f'{a}/{b}')


def average_numbers(numbers: list[int]):
    return np.average(numbers)

For the tests:

from urllib.parse import quote_plus
import pytest
import requests_mock

from my_pdm_project.mymath import (
    add_numbers,
    average_numbers,
    subtract_numbers,
    multiply_numbers,
    divide_numbers,
    make_mathjs_request,
    BASE_URI
)


def test_makemathjs_request():
    with requests_mock.Mocker() as m:
        m.get(f'{BASE_URI}{quote_plus("2+3")}', text='5')
        result = make_mathjs_request("2+3")
    assert result == 5
    assert type(result) is int


def test_add():
    with requests_mock.Mocker() as m:
        m.get(f'{BASE_URI}{quote_plus("2+3")}', text='5')
        assert add_numbers(2, 3) == 5


@pytest.mark.parametrize('x,y,expected', [(0, 3, -3), (5, 3, 2)])
def test_subtract(x, y, expected):
    with requests_mock.Mocker() as m:
        m.get(f"{BASE_URI}{quote_plus(f'{x}-{y}')}", text=f'{expected}')
        assert subtract_numbers(x, y) == expected


@pytest.mark.parametrize('x,y,expected', [(3, 0, 0), (2, 3, 6)])
def test_multiply(x, y, expected):
    with requests_mock.Mocker() as m:
        m.get(f"{BASE_URI}{quote_plus(f'{x}*{y}')}", text=f'{expected}')
        assert multiply_numbers(x, y) == expected


def test_divide():
    with requests_mock.Mocker() as m:
        m.get(f"{BASE_URI}{quote_plus(f'{6}/{3}')}", text='2')
        m.get(f"{BASE_URI}{quote_plus(f'{7}/{3}')}", text='2.3333333333333335')
        m.get(f"{BASE_URI}{quote_plus(f'{3}/{0}')}", text='')
        assert divide_numbers(6, 3) == 2
        assert divide_numbers(7, 3) == 2.3333333333333335
        with pytest.raises(ZeroDivisionError):
            divide_numbers(3, 0)


def test_average():
    assert average_numbers([90, 88, 99, 100]) == 94.25

Everything is now mocked up, but after running the tests an issue comes up:

expression = '7/3'

    def make_mathjs_request(expression: str):
        res = requests.get(
            f'{BASE_URI}{quote_plus(expression)}'
        )
>       return int(res.text)
E       ValueError: invalid literal for int() with base 10: '2.3333333333333335'

src/my_pdm_project/mymath.py:12: ValueError

The problem is that 2.3333333333333335 is not an integer value. Given the operations that division supports, we want to return a float data type instead which can handle numbers with a decimal point. Another issue is that we don't know how to handle the ZeroDivisionError case since MathJS is handling it for us instead of python now. The problem is we're only giving our MathJS calling method an expression for the input so it doesn't know what to do. To handle this we'll force it to require both the arguments and the operation. The end result code will look like this:

Note: The code here is meant to show how a more involved code refactor is handled and is by no means very practical logic wise. Python can handle such basic math functions as-is quite well enough, and you generally don't want to switch return types for a function.

from urllib.parse import quote_plus
import numpy as np
import requests

BASE_URI = "http://api.mathjs.org/v4/?expr="
SUPPORTED_OPERATIONS = ['+', '-', '*', '/']


def make_mathjs_request(a: int, b: int, operation: str):
    if operation == '/' and b == 0:
        raise ZeroDivisionError
    elif operation not in SUPPORTED_OPERATIONS:
        raise ValueError

    operation_expression = quote_plus(f'{a}{operation}{b}')
    res = requests.get(
        f'{BASE_URI}{operation_expression}'
    )

    if operation == '/':
        return float(res.text)
    else:
        return int(res.text)


def add_numbers(a: int, b: int):
    return make_mathjs_request(a, b, '+')


def subtract_numbers(a: int, b: int):
    return make_mathjs_request(a, b, '-')


def multiply_numbers(a: int, b: int):
    return make_mathjs_request(a, b, '*')


def divide_numbers(a: int, b: int):
    return make_mathjs_request(a, b, '/')


def average_numbers(numbers: list[int]):
    return np.average(numbers)

So this allows us to check for division by 0 and as a bonus I've established a list of supported operations to ensure we're working within the scope of what our module needs to do. If division is the operation we also return a float instead. Now for the tests:

from urllib.parse import quote_plus
import pytest
import requests_mock

from my_pdm_project.mymath import (
    add_numbers,
    average_numbers,
    subtract_numbers,
    multiply_numbers,
    divide_numbers,
    make_mathjs_request,
    BASE_URI
)


def test_makemathjs_request():
    with requests_mock.Mocker() as m:
        m.get(f'{BASE_URI}{quote_plus("2+3")}', text='5')
        m.get(f'{BASE_URI}{quote_plus("7/3")}', text='5')
        result = make_mathjs_request(2, 3, '+')
        result2 = make_mathjs_request(7, 3, '/')
    assert result == 5
    assert type(result) is int
    assert type(result2) is float


def test_makemathjs_unsupported_operation():
    with pytest.raises(ValueError):
        make_mathjs_request(2, 3, '~')


def test_add():
    with requests_mock.Mocker() as m:
        m.get(f'{BASE_URI}{quote_plus("2+3")}', text='5')
        assert add_numbers(2, 3) == 5


@pytest.mark.parametrize('x,y,expected', [(0, 3, -3), (5, 3, 2)])
def test_subtract(x, y, expected):
    with requests_mock.Mocker() as m:
        m.get(f"{BASE_URI}{quote_plus(f'{x}-{y}')}", text=f'{expected}')
        assert subtract_numbers(x, y) == expected


@pytest.mark.parametrize('x,y,expected', [(3, 0, 0), (2, 3, 6)])
def test_multiply(x, y, expected):
    with requests_mock.Mocker() as m:
        m.get(f"{BASE_URI}{quote_plus(f'{x}*{y}')}", text=f'{expected}')
        assert multiply_numbers(x, y) == expected


def test_divide():
    with requests_mock.Mocker() as m:
        m.get(f"{BASE_URI}{quote_plus(f'{6}/{3}')}", text='2')
        m.get(f"{BASE_URI}{quote_plus(f'{7}/{3}')}", text='2.3333333333333335')
        assert divide_numbers(6, 3) == 2
        assert divide_numbers(7, 3) == 2.3333333333333335
        with pytest.raises(ZeroDivisionError):
            divide_numbers(3, 0)


def test_average():
    assert average_numbers([90, 88, 99, 100]) == 94.25

The main change here is around the tests for our MathJS caller:

def test_makemathjs_request():
    with requests_mock.Mocker() as m:
        m.get(f'{BASE_URI}{quote_plus("2+3")}', text='5')
        m.get(f'{BASE_URI}{quote_plus("7/3")}', text='5')
        result = make_mathjs_request(2, 3, '+')
        result2 = make_mathjs_request(7, 3, '/')
    assert result == 5
    assert type(result) is int
    assert type(result2) is float


def test_makemathjs_unsupported_operation():
    with pytest.raises(ValueError):
        make_mathjs_request(2, 3, '~')

We're now testing if division returns a floating number and have updated to use the new parameter layout. We're also making sure an error is thrown if we're using an unsupported operation. Note that none of the other tests have really changed at all since we changed what happened in the function but not how the function was called. I removed the requests mock for 0 division since that will bail out early before we even attempt the API call. After a quick test run everything looks good and coverage is 100% as well:

$ pdm run pytest --cov=my_pdm_project  --cov-report term-missing
================================================================================== test session starts ==================================================================================
plugins: cov-4.1.0, requests-mock-1.11.0
collected 9 items

tests/test_mymath.py .........                                                                                                                                                     [100%]

---------- coverage: platform win32, python 3.11.5-final-0 -----------
Name                             Stmts   Miss  Cover   Missing
--------------------------------------------------------------
src/my_pdm_project/__init__.py       0      0   100%
src/my_pdm_project/mymath.py        25      0   100%
--------------------------------------------------------------
TOTAL                               25      0   100%

A final note here is that it's a good idea to increase our version number again since we've changed quite a lot with these updates. Go ahead and do so in pyproject.toml:

[project]
name = "my-pdm-project"
version = "0.3.0"

and then install again:

$ pdm install

Conclusion

Quite a lot was covered in this section! As I mentioned in the start testing is extremely valuable enough to where the article ended up longer than most. Having assurance that code works as intended even with substantial refactoring is a great feeling when you start working on more complex projects. Please feel free to spend as much time as you need here to get comfortable with testing as it's just that important.

Now up until now we've been mostly running our tooling separately. In the next installment we'll look at how we can package everything up in an orchestrated fashion to make things more streamlined.

Python Linting With Flake8 and Pylint

Chris White — Sun, 15 Oct 2023 02:58:54 +0000

Linting Basics
PEP 8
flake8
- flake8 Setup
- A Simple Run
- Making Fixes
- Long Lines And flake8 Configuration
Catching Coding Issues With pylint
Conclusion

In the last installment we put together a python project in an automated fashion using pdm. We also used to to manage the project itself through virtual environments and dependency management. Now the next improvement that we'll be working on is tools that can help us catch issues with our code.

Linting Basics

A linter is a program which performs a process called static code analysis on a codebase. Static code analysis is the process of reading in code into a structure that can be reasoned about based on certain rules. This can be anything from stylistic issues, to potential bugs, and even security issues. Linters also serve as a great tool when working in a collaborative environment to ensure a code quality baseline.

PEP 8

In terms of style guidelines for python, PEP8 is where most developers look. It's Python Enhancement Proposal (PEP) which proposes a style guideline for the language. PEP8 is great for situations where you plan to collaborate on your code as an open source project and want to decide on a style guideline that most will be familiar with. It's also the standard used for python standard library development. If you're working on a team as a professional developer however, there's a chance they have their own standards on how things work. As the PEP itself mentions always prioritize style guidelines from your team.

flake8

flake8 is a popular tool for managing PEP8 compliance for code. It also can detect a few common code issues that are outside of PEP8's scope. As there is a decent amount of usage of it among open source projects I highly recommend getting accustomed to it.

flake8 Setup

flake8 is something you have to install as a dependency. While technically it's a tool that you would expect to use in most of your projects, I would actually recommend installing it individually for each project instead of using pipx. This is because if you share your code with others having it as a dependency explicitly listed in the project makes it easier for others to install them. Now for pdm we're going to add flake8 to our project in a specific manner:

$ pdm add -dG dev flake8

This adds flake8 to a development group called "dev". When dealing with package installations there's generally two types:

Dev: Tooling included for code scanning, testing, etc. meant for working on development of the package
Prod: Meant to be as lightweight as possible to improve performance

By creating this separation production deployments will only contain the packages necessary to run the application and nothing more. pdm even includes options for handling the dev/prod separation:

$ pdm install --prod # production deployment with no dev dependencies
$ pdm install --dev # include dev dependencies

This also modifies the pyproject.toml by adding a new section:

[tool.pdm.dev-dependencies]
dev = [
    "flake8>=6.1.0",
]

One thing to note here is that pyproject.toml allows for tools to define their own properties via a tool declaration as shown. You'll start to see this more as you introduce new tools for dealing with python code.

A Simple Run

As adding a dependency also installs it in the virtual environment, we can run flake8 right away:

$ pdm run flake8

Chances are you got spammed with a lot of output. This is because the virtual environment contains python code for our dependencies. We want to avoid this since that's not a concern to the development of our project. We can mitigate this for now by running flake8 against src/ and tests/ exclusively:

$ pdm run flake8 src/ tests/
src/my_pdm_project/mymath.py:3:1: E302 expected 2 blank lines, found 1
src/my_pdm_project/mymath.py:6:1: E302 expected 2 blank lines, found 1
src/my_pdm_project/mymath.py:9:1: E302 expected 2 blank lines, found 1
src/my_pdm_project/mymath.py:12:1: E302 expected 2 blank lines, found 1
src/my_pdm_project/mymath.py:15:1: E302 expected 2 blank lines, found 1
tests/test_mymath.py:2:80: E501 line too long (114 > 79 characters)
tests/test_mymath.py:4:1: E302 expected 2 blank lines, found 1
tests/test_mymath.py:18:42: E231 missing whitespace after ','
tests/test_mymath.py:20:29: E231 missing whitespace after ','
tests/test_mymath.py:23:45: E231 missing whitespace after ','
tests/test_mymath.py:23:48: E231 missing whitespace after ','
tests/test_mymath.py:23:51: E231 missing whitespace after ','
tests/test_mymath.py:25:1: E305 expected 2 blank lines after class or function definition, found 1

Making Fixes

So we do have a few on our core file and some more on the test file we made. Let's take a look at the core file:

import numpy as np

def add_numbers(a: int, b: int):
    return a + b

def subtract_numbers(a: int, b: int):
    return a - b

def multiply_numbers(a: int, b: int):
    return a * b

def divide_numbers(a: int, b: int):
    return a / b

def average_numbers(numbers: list[int]):
    return np.average(numbers)

Most of the warnings here are from expected 2 blank lines, found 1. This is because PEP8 recommends "Surround top-level function and class definitions with two blank lines." which we're not doing here. I'll go ahead and do that:

import numpy as np


def add_numbers(a: int, b: int):
    return a + b


def subtract_numbers(a: int, b: int):
    return a - b


def multiply_numbers(a: int, b: int):
    return a * b


def divide_numbers(a: int, b: int):
    return a / b


def average_numbers(numbers: list[int]):
    return np.average(numbers)

Another run shows that flake8 is happy with the new changes:

pdm run flake8 .\src\ .\tests\
.\tests\test_mymath.py:2:80: E501 line too long (114 > 79 characters)
.\tests\test_mymath.py:4:1: E302 expected 2 blank lines, found 1
.\tests\test_mymath.py:18:42: E231 missing whitespace after ','
.\tests\test_mymath.py:20:29: E231 missing whitespace after ','
.\tests\test_mymath.py:23:45: E231 missing whitespace after ','
.\tests\test_mymath.py:23:48: E231 missing whitespace after ','
.\tests\test_mymath.py:23:51: E231 missing whitespace after ','
.\tests\test_mymath.py:25:1: E305 expected 2 blank lines after class or function definition, found 1

Long Lines And flake8 Configuration

Now it's time to deal with the test file:

import unittest
from my_pdm_project.mymath import add_numbers, average_numbers, subtract_numbers, multiply_numbers, divide_numbers

class TestMyMathMethods(unittest.TestCase):

    def test_add(self):
        self.assertEqual(add_numbers(2, 3), 5)

    def test_subtract(self):
        self.assertEqual(subtract_numbers(0, 3), -3)
        self.assertEqual(subtract_numbers(5, 3), 2)

    def test_multiply(self):
        self.assertEqual(multiply_numbers(3, 0), 0)
        self.assertEqual(multiply_numbers(2, 3), 6)

    def test_divide(self):
        self.assertEqual(divide_numbers(6,3), 2.0)
        with self.assertRaises(ZeroDivisionError):
            divide_numbers(3,0)

    def test_average(self):
        self.assertEqual(average_numbers([90,88,99,100]), 94.25)

if __name__ == '__main__':
    unittest.main()

The first complaint is that line 2 is too long. Due to how common it is to list out a number of imports like this, python established the ability to group them with parentheses in PEP328. So we can update the import like so:

from my_pdm_project.mymath import (
    add_numbers,
    average_numbers,
    subtract_numbers,
    multiply_numbers,
    divide_numbers
)

Now the line length of 79 characters is something that many projects may decide to diverge from with an override. The main reason for this listed in the PEP is "Limiting the required editor window width makes it possible to have several files open side by side, and works well when using code review tools that present the two versions in adjacent columns.". Now PEP8 does mention that the maximum line length can be adjusted to 99 at least. I'll go ahead and do this to show how flake8 can be configured. Create a .flake8 file in the project's root directory (where pyproject.toml is):

[flake8]
max-line-length = 99
exclude = .venv/*

The maximum line length will now be 99 and I also went ahead and used another setting which excludes our .venv directory so we can just run pdm run flake8 by itself. Now the next error is the same with the 2 blank lines before a function, just with the class definition case now:

)


class TestMyMathMethods(unittest.TestCase):

Next is a series of warnings about commas not having whitespace after them. I'll go ahead and make this simple change:

    def test_add(self):
        self.assertEqual(add_numbers(2, 3), 5)

    def test_subtract(self):
        self.assertEqual(subtract_numbers(0, 3), -3)
        self.assertEqual(subtract_numbers(5, 3), 2)

    def test_multiply(self):
        self.assertEqual(multiply_numbers(3, 0), 0)
        self.assertEqual(multiply_numbers(2, 3), 6)

    def test_divide(self):
        self.assertEqual(divide_numbers(6, 3), 2.0)
        with self.assertRaises(ZeroDivisionError):
            divide_numbers(3, 0)

    def test_average(self):
        self.assertEqual(average_numbers([90, 88, 99, 100]), 94.25)

This makes the arguments better separated visually. Finally is much like the two lines before the class definition, we also need two lines after it:

        self.assertEqual(average_numbers([90, 88, 99, 100]), 94.25)


if __name__ == '__main__':
    unittest.main()

This should fix everything so we'll go ahead and run flake8 once more:

$ pdm run flake8
$

This time there's no output, meaning no issues were found with our code.

Catching Coding Issues With pylint

Another tool I highly recommend is pylint. It tends to be more focused on fixing code related errors. As with flake8 we'll go ahead and install it as a dev dependency:

$ pdm add -dG dev pylint

Now much like flake8 we'll want to configure pylint for things like ignoring our virtual environment directory. One nice thing about pylint is that it can be configured through pyproject.toml. I'll go ahead and update it with a new configuration directive for pylint:

[project]
name = "my-pdm-project"
version = "0.1.0"
description = ""
authors = [
    {name = "Chris White", email = "me@cwprogram.com"},
]
dependencies = [
    "numpy>=1.25.2",
]
requires-python = ">=3.11"
readme = "README.md"
license = {text = "MIT"}

[build-system]
requires = ["pdm-backend"]
build-backend = "pdm.backend"

[tool.pdm.dev-dependencies]
dev = [
    "flake8>=6.1.0",
    "pylint>=3.0.1",
]

[tool.pylint.MASTER]
ignore-paths = [ "^.venv/.*$" ]

[tool.pylint."MESSAGES CONTROL"]
disable = '''
missing-module-docstring,
missing-class-docstring,
missing-function-docstring
'''

Now I also added something to ignore warnings about docstrings. That's because it's something I'd rather handle in a later installment once you're more comfortable with handling the existing linter warnings that might come up. It is also a nice way to showcase pylint's ability to disable certain linting issues if you feel you have a valid use case. Now pylint can be run like so:

$ pdm run pylint --recursive=y .

Now right now nothing shows up on our codebase, so I'm going to go ahead and adjust the mymath_script.py script we made from before to have a number of noticeable errors:

from my_pdm_project.mymath import add_numbers, nothing
import os

print(myvar)
myvar = 3

print(add_numbers(2, 3))

In this case I'll simply run pylint directly against the file:

$ pdm run pylint mymath_script.py
************* Module mymath_script
mymath_script.py:1:0: E0611: No name 'nothing' in module 'my_pdm_project.mymath' (no-name-in-module)
mymath_script.py:4:6: E0601: Using variable 'myvar' before assignment (used-before-assignment)
mymath_script.py:5:0: C0103: Constant name "myvar" doesn't conform to UPPER_CASE naming style (invalid-name)
mymath_script.py:2:0: C0411: standard import "import os" should be placed before "from my_pdm_project.mymath import add_numbers, nothing" (wrong-import-order)
mymath_script.py:2:0: W0611: Unused import os (unused-import)

Now to figure out what's going on with each of the messages here we can refer to pylint's messages overview page. Here I'll look at the first warning about no name 'nothing'. This is warning us that we're trying to import something that doesn't exist. This could either be a typo, or something that provides it should in fact exist. In this case it shouldn't be there at all so I'll go ahead and remove it:

from my_pdm_project.mymath import add_numbers
import os

print(myvar)
myvar = 3

print(add_numbers(2, 3))

Now there are two warnings about myvar. One is that it's used before assignment since print(myvar) is used before myvar is actually defined. Another issue is that myvar is not upper case. The reason why the message is showing is that myvar is considered a constant. As the name implies that's because the value is constant the whole time. Naming them in upper case is recommended as many other languages follow this convention and it makes the usage of it very clear. I'll go ahead and fix both issues now:

from my_pdm_project.mymath import add_numbers
import os

MYVAR = 3
print(MYVAR)

print(add_numbers(2, 3))

The final issue is with the os module. First is the wrong module import order. os is considered a "standard library module". The rule is that you want to be importing standard library modules before anything else. Putting it like this would fix the issue:

import os
from my_pdm_project.mymath import add_numbers

MYVAR = 3
print(MYVAR)

print(add_numbers(2, 3))

However the next message makes this change invalid since the other issue is we're not even using the os module in the first place. This situation frequently happens when standard library modules are brought in to debug code quickly. Removing the import line is good enough to solve this:

from my_pdm_project.mymath import add_numbers

MYVAR = 3
print(MYVAR)

print(add_numbers(2, 3))

After all the changes are made pylint shows us in the clear:

$ pdm run pylint mymath_script.py

-------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 6.00/10, +4.00)

Thanks to these linting tools our code quality baseline has been raised higher.

Conclusion

Linters are a great way to slowly understand about how things should be structured in python. If any of the linting errors seem confusing to you don't be afraid to ask around and see why you're getting the message. This will help improve your overall python knowledge and having linter messages as context is a great way to get targeted help. In the next installment we'll be looking at how to use testing to supplement our linter checks in making the code even more solid.

Beginning Python: Project Management With PDM

Chris White — Thu, 12 Oct 2023 13:09:32 +0000

Centralizing Python Package Commands With pipx
PDM
- Project Creation
- Adjusting Code and Building
- Interacting With The Virtual Environment
- Dependency Management
Conclusion

In the last installment we learned about the basics of how a python project was laid out. This time we're going to see how to automate the creation of a project with said layout. The goal is to have a system setup where a few commands can effortlessly produce a new python project.

Centralizing Python Package Commands With pipx

One of the issues that we'll have to deal with first is when we need to work with centralized python based programs. A majority of these programs are useful in improving our overall workflow efficiency. Some of these tools are best handled outside of the virtual environment so all projects can use them. To handle this we'll be using pipx. It will install python based programs in a way where they don't clutter up the system python but are also available to all virtual environments without further installation. In this case we'll go ahead and install pipx in a special manner, and we'll do so using our 3.11 version of python:

$ python3.11 -m pip install --user pipx

Now Windows requires a little bit more work. Simply run this command (this is assuming you're still working with the Python 3.11 from the last installment):

>  & $env:APPDATA\Python\Python311\Scripts\pipx ensurepath

To make sure Windows knows where to find pipx when it's called. You'll need to close and re-open your terminal for these changes to take effect. Once that's finished you should now be able to verify pipx works like so:

$ pipx --version
1.2.0

Now I'll go ahead and demonstrate how this works. Calling pipx is mostly the same as how you would call standard pip. In this case we'll install pycowsay. This is a python version of the vanity program cowsay, which produces ascii art of a cow saying (or thinking of) something. Its small footprint and minimal application design makes it work really well for testing purposes (such as right now). So I'll go ahead and install it:

$ pipx install pycowsay
$ pycowsay mooooo

  ------
< mooooo >
  ------
   \   ^__^
    \  (oo)\_______
       (__)\       )\/\
           ||----w |
           ||     ||

Now to really test that this works, I'll attempt to do it in the virtual environment created last time:

$ cd my-python-test
$ venv/bin/activate # .\venv\Scripts\activate.ps1 on Windows
(venv) my-python-test $ pycowsay mooooo

  ------
< mooooo >
  ------
   \   ^__^
    \  (oo)\_______
       (__)\       )\/\
           ||----w |
           ||     ||

Despite pycowsay having not been installed in the virtual environment, I'm able to run it without any issues. Since we're done with the virtual environment for demonstration purposes we can go ahead and deactivate it:

(venv) my-python-test $ deactivate
$

With the setup complete I'm going to introduce some tooling that can help with automation of generating proper python package layout. To give a refresh the current layout of our project looks like:

│   README.md
│   LICENSE
│   pyproject.toml
├───src
│   ├───my_python_test
│       └───__init__.py
│           mymath.py
│
├───tests
│   │───test_mymath.py
├───venv

PDM

PDM is a solution that allows for easy creation and management of python projects. Some of the key features that will improve the management of python projects include:

Automated generation of project layout including pyproject.toml
Creation of virtual environments
Building python packages
Management of dependencies
Project templates

So let's look at some of these features (save the project template one because I'll be making a dedicated article on that later).

Project Creation

First we'll need to install pdm. This can be done through pipx so we can use it for all our python projects:

$ pipx install pdm

Now we'll need to make a directory for the code so I'll go ahead and make a folder called my-pdm-project:

$ mkdir my-pdm-project
$ cd my-pdm-project

Now we can initialize a new python project through the following:

$ pdm init --python python3.11 --lib --backend pdm-backend --non-interactive

--python python3.11 ensures the main python version for the project is 3.11. --lib allows for a more filled out pyproject.toml file. Despite the name I recommend having this on in general since it produces a more useful directory layout. --backend pdm-backend is stating that pdm will be handling the building of our python code. In other words it will be a replacement for the build module we used in the last installment to build our python package. Finally --non-interactive skips interactive prompts that would normally be asked and instead utilizes sensible beginner defaults. After execution the following directory structure is created:

my-pdm-project
├── .gitignore
├── .pdm-python
├── pyproject.toml
├── README.md
├── tests
│   ├── __init__.py
├── src
│   └── my_pdm_project
│       ├── __init__.py
└── .venv

Comparing to the layout we had in the last installment there are a number of new files. pdm has created a virtual environment for us in the .venv directory. Another nice feature is that pdm also generates a python specific .gitignore file. This is useful for when you're sharing code through sites like GitHub. It will prevent various python related directories from being committed that shouldn't be (a virtual environment is one such example). The next is .pdm-python, which simply points to the python interpreter that's used in the project. In this case it will be the python that's part of the virtual environment pdm created for us. The virtual environment itself is present in the .venv directory. With the exception of the missing LICENSE file everything else from our previous layout is the same. Let's take a look at the pyproject.toml file as well:

[project]
name = "my-pdm-project"
version = "0.1.0"
description = ""
authors = [
    {name = "Chris White", email = "me@no.spam"},
]
dependencies = []
requires-python = ">=3.11"
readme = "README.md"
license = {text = "MIT"}

[build-system]
requires = ["pdm-backend"]
build-backend = "pdm.backend"

now to compare against the one we hand made:

[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"

[project]
name = "my-python-test"
version = "0.0.1"
authors = [
  { name="Chris White", email="no@spam.com" },
]
description = "A small example package"
readme = "README.md"
requires-python = ">=3.9"
classifiers = [
    "Programming Language :: Python :: 3",
    "License :: OSI Approved :: MIT License",
    "Operating System :: OS Independent",
]

As mentioned the build system is different from what we used previously, setuptools. Classifiers is the main portion that is missing. This is a way to categorize the python code like a kind of labeling system and we'll look at that in a later installment in the series.

Adjusting Code and Building

It's time to pull in the code we used last time. Just the tests will need a slight rename modification in the code:

src/my_pdm_project/mymath.py

def add_numbers(a: int, b: int):
    return a + b

def subtract_numbers(a: int, b: int):
    return a - b

def multiply_numbers(a: int, b: int):
    return a * b

def divide_numbers(a: int, b: int):
    return a / b

tests/test_mymath.py

import unittest
from my_pdm_project.mymath import add_numbers, subtract_numbers, multiply_numbers, divide_numbers

class TestMyMathMethods(unittest.TestCase):

    def test_add(self):
        self.assertEqual(add_numbers(2, 3), 5)

    def test_subtract(self):
        self.assertEqual(subtract_numbers(0, 3), -3)
        self.assertEqual(subtract_numbers(5, 3), 2)

    def test_multiply(self):
        self.assertEqual(multiply_numbers(3, 0), 0)
        self.assertEqual(multiply_numbers(2, 3), 6)

    def test_divide(self):
        self.assertEqual(divide_numbers(6,3), 2.0)
        with self.assertRaises(ZeroDivisionError):
            divide_numbers(3,0)

if __name__ == '__mai

Now we can build and install this code in one simple step:

> pdm install
Lock file does not exist
Updating the lock file...
🔒 Lock successful
Changes are written to pdm.lock.
All packages are synced to date, nothing to do.
Installing the project as an editable package...
  ✔ Install my-pdm-project 0.1.0 successful

🎉 All complete!

Don't worry about the mentioned lock file for now, we'll get to that in a bit. Normally the resulting files for building go in a dist directory but this time they're in a .pdm-build directory. Looking inside we don't see the .tar.gz or .whl files like we did before:

.pdm-build
├── .gitignore
├── my_pdm_project.pth
└── my_pdm_project-0.1.0+editable.dist-info
    ├── METADATA
    └── WHEEL

While the files expected are missing there is a .pth file. Looking at the contents:

/home/johndoe/my-pdm-project/src

It's instead pointing to the src folder in our project where our python code resides. This is a special type of installation called an "editable install". Instead of having to package our code and install every single time, we instead get a live view of the code that python can use. This is ideal in development environments where code is constantly changing. It also allows for easier testing as we don't have to go into the src/ directory anymore to test. If you'd rather have the sdist and wheel formats instead you can run pdm install with the --no-editable argument:

$ pdm install --no-editable

Another option is to simply build our package but not install it:

$ pdm build

Much like the build module from last time, it will build both an sdist and wheel then put them in the dist directory:

dist
├── my_pdm_project-0.1.0-py3-none-any.whl
└── my_pdm_project-0.1.0.tar.gz

Interacting With The Virtual Environment

The virtual environment that pdm creates has a few methods of interaction. You can enter it as a standard virtual environment:

Linux

$ source .venv/bin/activate

Windows

 > .\.venv\Scripts\activate.ps1

or another option is to use pdm run. This will run the command following it in the context of the virtual environment. It's a nice replacement for one time actions where you'd rather not have to activate the virtual environment, run the command, then deactivate or leave it open. A great example of this is if you want to run unit tests:

$ pdm run python -m unittest
....
----------------------------------------------------------------------
Ran 4 tests in 0.000s

OK

This is also useful if you're writing python scripts for your code and need to run them in the virtual machine where the code is installed:

mymath_script.py

from my_pdm_project.mymath import add_numbers

print(add_numbers(2, 3))

$ pdm run python mymath_script.py
5

One interesting thing about the virtual environment that pdm creates is that it does not have pip installed:

$ source .venv/bin/activate
(venv) $ python -m pip
python: No module named pip

This is done on purpose to keep the resulting environment lightweight. If pip is missing though how do we install things for our project? That's where pdm's dependency management comes in.

Dependency Management

A majority of software in the modern world is built upon various third party packages. These packages help offload work that would otherwise be rather tedious. This includes interacting with cloud APIs, developing scientific applications, or even creating web applications. As you gain experience in python you'll be using more and more of these packages developed by others to power your own code. In this example I've decided to expand our math functionality with NumPy. pdm add is what's used to add dependencies like this to our project:

$ pdm add numpy
Adding packages to default dependencies: numpy
🔒 Lock successful
Changes are written to pyproject.toml.
Synchronizing working set with resolved packages: 1 to add, 0 to update, 0 to remove

  ✔ Install numpy 1.25.2 successful
Installing the project as an editable package...
  ✔ Update my-pdm-project 0.1.0+editable -> 0.1.0 successful

🎉 All complete!

To test this out I'll add a function to our mymath.py code which calculates averages:

import numpy as np

def add_numbers(a: int, b: int):
    return a + b

def subtract_numbers(a: int, b: int):
    return a - b

def multiply_numbers(a: int, b: int):
    return a * b

def divide_numbers(a: int, b: int):
    return a / b

def average_numbers(numbers: list[int]):
    return np.average(numbers)

And a small test added:

import unittest
from my_pdm_project.mymath import add_numbers, average_numbers, subtract_numbers, multiply_numbers, divide_numbers

class TestMyMathMethods(unittest.TestCase):

    def test_add(self):
        self.assertEqual(add_numbers(2, 3), 5)

    def test_subtract(self):
        self.assertEqual(subtract_numbers(0, 3), -3)
        self.assertEqual(subtract_numbers(5, 3), 2)

    def test_multiply(self):
        self.assertEqual(multiply_numbers(3, 0), 0)
        self.assertEqual(multiply_numbers(2, 3), 6)

    def test_divide(self):
        self.assertEqual(divide_numbers(6,3), 2.0)
        with self.assertRaises(ZeroDivisionError):
            divide_numbers(3,0)

    def test_average(self):
        self.assertEqual(average_numbers([90,88,99,100]), 94.25)

if __name__ == '__main__':
    unittest.main()

Now to go ahead an make sure everything is fine by running our tests:

$ pdm run python -m unittest
.....
----------------------------------------------------------------------
Ran 5 tests in 0.000s

OK

Now a few things have happened with our project that enabled this functionality. First off is pyproject.toml got an update:

[project]
name = "my-pdm-project"
version = "0.1.0"
description = ""
authors = [
    {name = "Chris White", email = "me@cwprogram.com"},
]
dependencies = [
    "numpy>=1.25.2",
]
requires-python = ">=3.11"
readme = "README.md"
license = {text = "MIT"}

[build-system]
requires = ["pdm-backend"]
build-backend = "pdm.backend"

As you can see a new dependencies section has been added with numpy included. We can also see what's going on with the mysterious lock file. This file is present as pdm.lock and the contents for me looks like this (contents may vary as newer versions of numpy are released):

# This file is @generated by PDM.
# It is not intended for manual editing.

[metadata]
groups = ["default"]
cross_platform = true
static_urls = false
lock_version = "4.3"
content_hash = "sha256:605a425b5a013be2d914d4cd0ceb7048e9f3923724a12f97187982c4b10a96dd"

[[package]]
name = "numpy"
version = "1.25.2"
requires_python = ">=3.9"
summary = "Fundamental package for array computing in Python"
files = [
    {file = "numpy-1.25.2-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:c5462d19336db4560041517dbb7759c21d181a67cb01b36ca109b2ae37d32418"},
    {file = "numpy-1.25.2-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:c5652ea24d33585ea39eb6a6a15dac87a1206a692719ff45d53c5282e66d4a8f"},
    {file = "numpy-1.25.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0d60fbae8e0019865fc4784745814cff1c421df5afee233db6d88ab4f14655a2"},
    {file = "numpy-1.25.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:60e7f0f7f6d0eee8364b9a6304c2845b9c491ac706048c7e8cf47b83123b8dbf"},
    {file = "numpy-1.25.2-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:bb33d5a1cf360304754913a350edda36d5b8c5331a8237268c48f91253c3a364"},
    {file = "numpy-1.25.2-cp311-cp311-win32.whl", hash = "sha256:5883c06bb92f2e6c8181df7b39971a5fb436288db58b5a1c3967702d4278691d"},
    {file = "numpy-1.25.2-cp311-cp311-win_amd64.whl", hash = "sha256:5c97325a0ba6f9d041feb9390924614b60b99209a71a69c876f71052521d42a4"},
    {file = "numpy-1.25.2-pp39-pypy39_pp73-macosx_10_9_x86_64.whl", hash = "sha256:1a1329e26f46230bf77b02cc19e900db9b52f398d6722ca853349a782d4cff55"},
    {file = "numpy-1.25.2-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4c3abc71e8b6edba80a01a52e66d83c5d14433cbcd26a40c329ec7ed09f37901"},
    {file = "numpy-1.25.2-pp39-pypy39_pp73-win_amd64.whl", hash = "sha256:1b9735c27cea5d995496f46a8b1cd7b408b3f34b6d50459d9ac8fe3a20cc17bf"},
    {file = "numpy-1.25.2.tar.gz", hash = "sha256:fd608e19c8d7c55021dffd43bfe5492fab8cc105cc8986f813f8c3c048b38760"},
]

So what's happening is that pdm has snapshot the version of numpy we installed at the time, with various references to different installation options (mostly different operating systems in this case). Why would we want the version snapshot like this? This process is actually fairly common in the development world and is referred to as "version pinning". The idea is that our underlying libraries we use are updated at some frequency. What if bug is introduced into our code that causes things to break? To deal with this we pin our version so we know that everything should work the same every time we run it. You'll likely see this as a professional developer to keep production systems running with consistency. If you do need to update a package, the process would look something like this:

pdm update numpy
Update code if necessary
Run your tests to make sure everything looks good

if something does break you've at least averted a disaster. One thing to keep in mind is that for maintainability reasons if you ever share this code you want to include the pdm.lock file as part of your codebase. This ensures that your users and contributors are working with the same dependencies as you are. Note that while it does mean the dependencies are the same, you're not guaranteed something could go wrong with other users. They might have a different operating system, or system configuration than you which causes something to break. The goal here is simply to reduce chances of different dependency versions being the issue!

pdm can also be used to remove dependencies. Let's say I install the SciPy package to make the math code even more amazing:

$ pdm add scipy

Then I realize that trying to maintain both math and science code at the same time probably isn't a good idea. I can just remove it like so:

$ pdm remove scipy

This will uninstall it from the virtual environment, remove it from pyproject.toml's dependency listing, and remove entries in the pdm.lock file. Now we have a fairly maintainable setup for working with our python code.

Conclusion

I encourage you to try out what's been discussed here, making a few projects along the way until you're more comfortable. You should also be in a fairly good spot to start developing out python code. Once you're more comfortable with managing your projects we'll start to look at some tooling that can help you catch issues with your code. See you in the next installment!