Forem: Chris White

The Great DNS Trail

Chris White — Mon, 27 Apr 2026 01:55:40 +0000

DNS or domain naming system is commonly utilized to allow websites the ability to be represented by a specific name. We'll be looking at the underlying parts of DNS in some detail to see much of what is normally abstracted away.

IP Addresses

In order to understand DNS, it's important to first understand what DNS is mapped to. A domain name by itself can't do anything which is why DNS exists in the first place. The actual means of accessing another system is done through IP addresses. As an example if I ping the dev.to site the address 151.101.66.217 comes back.

Now on the low level IP addresses are a sequences of bytes and the .s are mostly for visual purposes acting much like a comma in a sequences of numbers. If you were to use 32 bit binary the address 151.101.66.217 would come back as 10010111011001010100001011011001. Unlike domain names IP addresses are very compact and streamlined for quick network transport.

IP addresses are also bound by a numbering authority ICANN (Internet Corporation for Assigned Names and Numbers). This authority handles registration of specific IP ranges to corporations. 151.101.66.217 is part of the IP range 151.101.0.0 - 151.101.255.255 which according to ICANN's lookup belongs to the CDN (Content Distribution Network) provider Fastly. Note that there are special ranges of IPs known as private IPs. These sit behind routers and firewalls providing local network functionality.

Hosts File

Now before even touching DNS there's a quick check against what's known as the hosts file. On most systems this lies at /etc/hosts and windows it can be found at C:\Windows\System32\drivers\etc\hosts. The file looks something like this:

# localhost name resolution is handled within DNS itself.
#   127.0.0.1       localhost
#   ::1             localhost
192.168.1.91 rpi
192.168.1.93 rpi2
# End of section

This one is a mapping of my raspberry pi devices to their respective IP addresses on my local network. Instead of typing the IP address I can simply type rpi when SSH-ing in. Unlike actual domain names host file entries are more free form. Some advanced uses have domain names mapped to 0.0.0.0 providing a rudimentary domain blacklist.

DNS Cache

There's another entry point before DNS infrastructure is actually involved and that is DNS cache. Operating systems will vary in how they handle cache. Linux in most cases doesn't have DNS caching on by default due to its need to support a number of different scenarios. On Windows the cache may look something like this:

www.google-analytics.com
----------------------------------------
Record Name . . . . . : www.google-analytics.com
Record Type . . . . . : 1
Time To Live  . . . . : 9
Data Length . . . . . : 4
Section . . . . . . . : Answer
A (Host) Record . . . : 142.250.178.14

Browsers can also have their own built in DNS due to the large number of requests they typically make. Here's an example of my Firefox entry for dev.to:

From there caching works up several levels including routers, ISP/public servers, all the way up the chain.

Domain Name Authorities

Domain names have a hierarchical system of authorities starting at what are known as the root servers all the way down to the authoritative server that owns the domain name. This server is generally the registrar used to purchase the domain name.

IANA (Internet Assigned Numbers Authority) is a part of an ICANN affiliate which handles root DNS servers. These servers are the absolute authority on what are known as the top level domains. They keep a listing along with the entity who is authorized to represent the top level domain. For example, VeriSign Global Registry Service is the entity which is authorized to register domain names for the .com top level domain:

.com    generic     VeriSign Global Registry Services

The system also distinguishes between such generic top level domains and ones designated for countries:

.us     country-code    Registry Services, LLC

Most companies will go through the generic top level domain program if they themselves wish to act as an authority. There's also another process for top level domains representing countries. For those who wish to act as a domain registrar instead of a top level authority the accredited registrar program is available. Given how critical DNS infrastructure is all of these applications are strictly vetted so it's certainly a large investment.

The DNS Flow

So assuming the answer to a query of what something like dev.to is mainly starts at the locally designated DNS servers. This will most likely be your ISP but it's not uncommon to utilize public DNS servers such as Quad9, Google, and Cloud Flare. Public DNS servers were popularized after "certain ISPs" decided that showing ad littered pages for unknown domains was a good idea.

So let's say I'm looking for dev.to. Assuming it isn't cached my designated DNS server will reach out to one of the root DNS servers (h.root-servers.net for example) to get information on .to's ownership. .to is country code tld managed by the Kingdom of Tonga. It's not uncommon for country based TLDs to be used for vanity purposes such as .ai. The .to authority (ns01.trs-dns.com as an example) will then delegate the request to the authoritative nameservers which in this case are owned by Cloud Flare who would be considered to be the registrar that dev.to worked with to obtain (or transfer) the domain (josh.ns.cloudflare.com for example). This will then provide a number of responses:

;; ANSWER SECTION:
│dev.to.                 300     IN      A       151.101.194.217
│dev.to.                 300     IN      A       151.101.130.217
│dev.to.                 300     IN      A       151.101.66.217
│dev.to.                 300     IN      A       151.101.2.217

Any of these addresses will allow access to the dev.to site. Sometimes the entry comes back as a CNAME which is essentially an alias to another domain. In this case that will form a new request for the user's designated DNS server. All of these responses will be cached appropriately to ensure timely return to the client.

It's important to note that the user's DNS query to the ISP/Public DNS server is what's known as a recursive query as the server is handling the entire chain for you. The process of the ISP/Public DNS server going root server -> tld server -> authoritative server is known as incremental lookup. Note that if you attempt to query a root name server (recursive by default) it will simply let you know it doesn't support recursive calls and return the incremental version instead.

DNS Packets

DNS utilizes UDP (user datagram protocol) instead of the traditional TCP/IP used by many applications. The UDP protocol is extremely simple with fields for source port, destination port, length of packet, checksum, and data. This avoids the overhead of many of TCP's features which means packets would take longer to return. UDP is generally wrapped around one of the IP protocols to indicate the source and destination IP. DNS packet data is built around messages as defined in RFC 1035. By using the dig DNS tool we can see some of what that looks like:

$ dig dev.to
│
│; <<>> DiG 9.20.21-1~deb13u1-Debian <<>> dev.to
│;; global options: +cmd
│;; Got answer:
│;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41311
│;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1
│
│;; OPT PSEUDOSECTION:
│; EDNS: version: 0, flags:; udp: 4096
│;; QUESTION SECTION:
│;dev.to.                                IN      A
│
│;; ANSWER SECTION:
│dev.to.                 261     IN      A       151.101.194.217
│dev.to.                 261     IN      A       151.101.66.217
│dev.to.                 261     IN      A       151.101.2.217
│dev.to.                 261     IN      A       151.101.130.217
│
│;; Query time: 0 msec
│;; SERVER: 192.168.1.254#53(192.168.1.254) (UDP)
│;; WHEN: Sun Apr 26 20:17:30 EDT 2026
│;; MSG SIZE  rcvd: 99

Here I'm doing a recursive query directed at my router (which is pointed to Google's DNS servers). The router has indicated it's a server that is capable of doing recursive queries. The answer section is the resulting IPs behind dev.to. Now let's compare this to another query:

$dig @ns01.trs-dns.com dev.to
│
│; <<>> DiG 9.20.21-1~deb13u1-Debian <<>> @ns01.trs-dns.com dev.to
│; (2 servers found)
│;; global options: +cmd
│;; Got answer:
│;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 61237
│;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 2, ADDITIONAL: 1
│;; WARNING: recursion requested but not available
│
│;; OPT PSEUDOSECTION:
│; EDNS: version: 0, flags:; udp: 1232
│;; QUESTION SECTION:
│;dev.to.                                IN      A
│
│;; AUTHORITY SECTION:
│dev.to.                 900     IN      NS      josh.ns.cloudflare.com.
│dev.to.                 900     IN      NS      jill.ns.cloudflare.com.
│
│;; Query time: 91 msec
│;; SERVER: 2620:57:4001::1#53(ns01.trs-dns.com) (UDP)
│;; WHEN: Sun Apr 26 20:03:51 EDT 2026
│;; MSG SIZE  rcvd: 118

In this case dig is letting me know that the server doesn't support recursive queries and I'm given an incremental result instead. Because I'm not given the actual records for the domain in question the "AUTHORITY" section is populated instead pointing me to the authoritative nameservers. When the actual DNS records for the domain name comes back the "ANSWER" records are populated instead.

DNS Records

There are a lot of DNS record types. Which of these will be acted upon will depend on the client accessing the information. A good majority of the time the important records you'll be working with are:

A: IPv4 address backing the domain
AAA: IPv6 address backing the domain
CNAME: points to another host to do DNS lookup for
MX: Email server backing the domain's email services
TXT: Freeform field used for a number of purposes
NS: The authoritative name servers for the domain

A and AA records are the most vital in obtaining the IP address to map to a domain name in a request.

DNSSEC

Being a simple plain text protocol DNS has security issues. If a malicious actor was able to spoof a DNS response that response would be cached among the global DNS infrastructure. This is commonly known as DNS Spoofing or DNS Cache Poisoning.

In an attempt to deal with this DNSSEC was proposed. The system works off of public key cryptography which is the same technology that powers SSL certs. DNS servers use cryptographic signatures to sign their data which are validated by their parent authority (.to's owner in the case of dev.to). The exception being the root DNS servers who have no parent authority. It's why the private keys for the root servers are very well guarded.

It's important to note that DNSSEC support is not mandatory as of right now. Though it's of course recommended to utilize it whenever possible.

Internal DNS

What I've described until now is for the public facing side. Companies are also able to utilize DNS for their own internal domain name resolution. This is done by pointing company equipment at an internal DNS server (or updating network configs for VPN users). The internal server will resolve for internal hosts and generally acts as a proxy if the domain is global such as google.com. Such servers are often hosted with software such as BIND9 on *NIX based systems and the DNS components that are part of Windows Server installs.

Wrap Up

Thanks for joining me in this very long look at the workings of DNS. I hope this proves to be an enlightening experience for those looking to deepen their knowledge on how systems work. It might be useful if you need to debug DNS issues as well!

Open Source And The Contribution Cemetery

Chris White — Fri, 24 Apr 2026 15:59:31 +0000

As someone has appreciates the value of open source, I'd like to touch on a rather troubling issue I've found. An issue which has been further made problematic with the recent explosion of AI. It's something I refer to as the contribution cemetery.

Thar be dragons here

I'm going to show you some stats for a few projects on GitHub.

python requests 145 issues 80 pull requests
vscode 5k+ issues 1.8k pull requests
kubernetes 1.8k issues 889 pull requests
Ruby On Rails 518 issues 1.1k pull requests
Rust Language 5k+ issues 1.1k pull requests
lodash 68 issues 72 pull requests

lodash, a popular NPM module, has what seems to be a small amount. However, 68 issues is quite a lot to go through. This is a fairly common pattern among popular open source software.

The burden of maintenance

While the code for open source projects is free and available, the cost of maintaining it is still very real. Python requests currently has 2 maintainers, though it used to have more. Some projects, such as kubernetes, have a decent amount of individuals handling the project. Yet still we have these fairly large amounts of issues and PRs. To make matters worse, AI powered PRs have caused further stress on maintaining a popular open source project. Even the Linux Kernel is starting to remove code that isn't well maintained due to the onslaught of security reports.

Just recently I was even going to install lunarvim, a vim distribution I used a while back, only to find out it was abandoned.

Contributor dilemma

So let's say there's someone who wants to contribute to one of these projects. Even if there was something to fix they would need to make sure it didn't conflict with several hundred/thousand PRs and figure out which of several hundred/thousand issues might cover it. Given the huge popularity of GitHub within development communities this seems to almost be a "if everyone can contribute no one can contribute" type situation.

I'd also say it can be difficult to find projects to contribute to if you're looking for something smaller in popularity. The bigger question is if they actually need help or if it's just someone who wants to share code for educational purposes and has no interest in maintaining it. While GitHub does have a collections page categorizing certain repositories, I've found that many of them fall under the same issue/pr ratio.

Where to start

As far as what to do about the issue it's pretty difficult. Most of the solutions will vary depending on how big the community is.

Bug wrangling

My first exposure to open source was the Gentoo Linux project. I spent a lot of time there doing something called "bug wrangling". Essentially I would go into bug reports, try to reproduce the issue (I had access to a lot of architectures), and try to point the maintainer in the right direction as to where the problem might be. Something like this on a larger scale could help bring issues to a more reasonable count. Larger projects may need to do it over several periods to prevent too much volatility being pushed to users.

Bug report time outs

While certainly not ideal, I've seen many projects have timers on when an issue will expire. Thought I'm not sure if that could easily be retrofit. Some projects may benefit from slightly shorter bug report timeouts.

Project restructure

Sometimes it's simply the scale of the project. The Linux kernel, for example, supports several years worth of hardware. This monolithic design naturally increases how many issues and PRs are required to support it. Most projects would require gradual restructuring as an all and one approach would be too risky. It may also be a case of what constitutes the project vision and if they need to be more strict with it in terms of what PRs to accept.

Full reset

This would essentially be closing everything and starting from scratch. The main issue is doing it at scale and if everything would simply pile up again in numbers. Some contributors also might not be too thrilled seeing their work was closed out. There's also the question of if this could be done at scale.

New tooling

Utilize a different system, such as bugzilla, to track issues. PRs would be looked at only if they had an entry in the new project issue tracker. Having an SSO login that could utilize GitHub's authentication would help alleviate the pain of having to go to a separate site for issues. It could also be something that layers on top of GitHub issues/PRs with an easier to digest UI

Team expansion

Expand project teams to include people who just handle issue cleanup, more maintainers to approve PRs, expand linting to catch common issues, etc. It might be a good idea to have some form of a mentoring program. This would allow frequent contributors to help potential new contributors know how the project's particular development process works.

What the future brings

While this has certainly been an issue for a while, I believe that the recent surge in AI adoption means it's a problem that should be considered sooner rather than later. While open source is certainly far from dead, stagnation due to a surge of contributions isn't very healthy. I hope it's something the community as a whole starts seriously looking into.

A Quick Look At The Proc Filesystem

Chris White — Thu, 23 Apr 2026 02:17:03 +0000

When looking through the filesystem of a Linux system you may notice a directory named /proc. It's a fascinating directory which exposes many of the internal data for the kernel. I'd like to show some of the interesting information you can get from /proc as well as some practical applications in popular software.

Finding One's Self

One of the more interesting pieces of information you can find is for the current process located in /proc/self:

cwprogram@rpi:/proc $ ls -lah /proc/self/
total 0
dr-xr-xr-x   9 cwprogram cwprogram 0 Apr 22 20:49 .
dr-xr-xr-x 217 root      root      0 Dec 31  1969 ..
dr-xr-xr-x   2 cwprogram cwprogram 0 Apr 22 20:49 attr
-rw-r--r--   1 cwprogram cwprogram 0 Apr 22 20:49 autogroup
-r--------   1 cwprogram cwprogram 0 Apr 22 20:49 auxv
-r--r--r--   1 cwprogram cwprogram 0 Apr 22 20:49 cgroup
--w-------   1 cwprogram cwprogram 0 Apr 22 20:49 clear_refs
-r--r--r--   1 cwprogram cwprogram 0 Apr 22 20:49 cmdline
-rw-r--r--   1 cwprogram cwprogram 0 Apr 22 20:49 comm
-rw-r--r--   1 cwprogram cwprogram 0 Apr 22 20:49 coredump_filter
-r--r--r--   1 cwprogram cwprogram 0 Apr 22 20:49 cpuset
lrwxrwxrwx   1 cwprogram cwprogram 0 Apr 22 20:49 cwd -> /proc
-r--------   1 cwprogram cwprogram 0 Apr 22 20:49 environ
lrwxrwxrwx   1 cwprogram cwprogram 0 Apr 22 20:49 exe -> /usr/bin/ls
<truncate>

Due to ls being the current process when the listing is run information for it is made available. There's also a cwd which points to the current working directory and exe that points to the executable. There's a status file available with a decent amount of information as well:

cwprogram@rpi:/proc $ cat self/status
Name:   cat
Umask:  0002
State:  R (running)
Tgid:   9755
Ngid:   0
Pid:    9755
PPid:   9687

You can use this with a bit of basic grep to get names of processes like so:

cwprogram@rpi:/proc $ grep "Name:" */status
102/status:Name:        kworker/0:1H-kblockd
1124/status:Name:       agetty
1126/status:Name:       agetty
11/status:Name: kworker/u16:0-ipv6_addrconf
131/status:Name:        kworker/R-mmc_complete
<truncate>

The first part of the path here is the PID itself. So this gives you a somewhat rudimentary process listing. Granted it's certainly not as user friendly as the ps command.

Finding Mounts

Processes also have mount information available in either a mountinfo or mounts file. The former is a bit more detailed:

$ cat mountinfo
20 25 0:19 / /sys rw,nosuid,nodev,noexec,relatime shared:6 - sysfs sysfs rw
21 25 0:20 / /proc rw,relatime shared:11 - proc proc rw
22 25 0:6 / /dev rw,nosuid,relatime shared:2 - devtmpfs udev rw,size=1672900k,nr_inodes=418225,mode=755

And the later may be a more familiar output:

cwprogram@rpi:/proc/self $ cat mounts
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,relatime 0 0
udev /dev devtmpfs rw,nosuid,relatime,size=1672900k,nr_inodes=418225,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=600,ptmxmode=000 0 0

This is for the process itself keeping namespace restrictions into consideration.

Topping It Off

Outside of the standard process information, /proc also has a number of toplevel files with interesting entries in them:

devices - Listing of character and block devices
meminfo - Memory statistics
mounts - Similar to the process version, except for the top system level
crypto - Various crypto ciphers available to the system
stat - Several statistics about the system
version - Kernel version string
cmdline - The options given to the kernel at boot
filesystems - Filesystems available to the kernel
cgroups - Cgroup information, particularly of use to container based solutions

Many of these contain useful information for the case of debugging fairly stripped down environments commonly found in container operating systems.

The Programmatic Approach

Now this isn't just for operating system debugging. It also has practical uses in modern day software. Take for example kubernetes:

            cmdline, err := os.ReadFile(filepath.Join("/proc", entry.Name(), "cmdline"))
            if err != nil {
                klog.V(4).Infof("Error reading file %s: %+v", filepath.Join("/proc", entry.Name(), "cmdline"), err)
                continue
            }

Source

In this particular case kubernetes is using the proc system to obtain PIDs which match a specific regex. It does so by getting the command name from cmdline on the process directory level. AWS's firecracker VM which is used to power some of its services such as Lambda also uses /proc to obtain cgroup directory info from /proc/mounts:

        // search PROC_MOUNTS for cgroup mount points
        let f = File::open(proc_mounts_path)
            .map_err(|err| JailerError::FileOpen(PathBuf::from(proc_mounts_path), err))?;

        // Regex courtesy of Filippo.
        // This will match on each line from /proc/mounts for both v1 and v2 mount points.
        //
        // /proc/mounts cointains lines that look like this:
        // cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0
        // cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
        //
        // This Regex will extract:
        //      * "/sys/fs/cgroup/unified" in the "dir" capture group.
        //      * "2" in the "ver" capture group as the cgroup version taken from "cgroup2"; for v1,
        //        the "ver" capture group will be empty (len = 0).
        //      * "[...],relatime,cpu,cpuacct" in the "options" capture group; this is used for
        //        cgroupv1 to determine what controllers are mounted at the location.
        let re = Regex::new(
            r"^([a-z2]*)[[:space:]](?P<dir>.*)[[:space:]]cgroup(?P<ver>2?)[[:space:]](?P<options>.*)[[:space:]]0[[:space:]]0$",
        ).map_err(JailerError::RegEx)?;

Source

Cython, which is the official C implementation of the Python programming language also uses /proc for a few things, one of them includes obtaining the parent process ID of a process:

snprintf(stat_path, sizeof(stat_path), "/proc/%d/stat", (int)pid);

The stat file is a pretty cryptic thing to look at which gives somewhat more parser friendly statistics for a process:

9687 (bash) S 9686 9687 9687 34817 9994 4194304 49314 139925 0 3 104 36 299 142 20 0 1 0 43945508 9326592 1490 18446744073709551615 367249391616 367250706224 549003478784 0 0 0 65536 3686404 1266761467 1 0 0 17 3 0 0 0 0 0 367250815728 367250867132 367940014080 549003480647 549003480653 549003480653 549003481070 0

Source

The first entry is the process ID, the second the command, the third entry is the process state, and the fourth entry is the parent process ID of the process. While using C for tokenized parsing such as this is a bit awkward it does get the job done.

Wrapping It Up

This is just a small peak into the usefulness that is the proc filesystem. As mentioned it's great when you need a source of information for debugging a Linux based system. It's also useful for handling certain task programmatically, especially if you're doing any form of container development. I urge you to look around /proc some more to see what other useful things you can find.

Security and String Interpolation

Chris White — Thu, 16 Apr 2026 23:41:25 +0000

I've come back to share knowledge again after being in a long hiatus. Though it's more of a "back to the job searching" situation at the moment. I've recently been looking over recent CVEs and thought I'd share some knowledge of a rather common source of vulnerabilities: string interpolation.

What Is String Interpolation?

This is a common feature of most programming languages which is a string containing special values which are then interpreted by the appropriate runtime. Here's a simple example in python:

input_string = "Hello, World"
print(f"Variable expanded to: {input_string}")

This would output Variable expanded to: Hello, World when run (well unless you're running a fairly old version of python). These are referred to as formatted string literals or f-strings. In this case the {input_string} is a placeholder which is taken by the python runtime and expanded out to the "Hello, World" string. Now a more practical use of this is taking user content and interacting with systems in a dynamic way. Unfortunately, not every user has good intentions.

String Interpolation Security

Given how widely used string interpolation is there's several categories of them you can find. The differences tend to be what the overall impact of a successful exploitation is. In most cases these exploits can be prevented by using context specific functions to cleanup any potential malicious content.

The Infamous eval

eval in many languages is a function that will evaluate the contents of a string (though certain languages may accept other objects). The main security issue with it is there's no real context and it's unrestricted for the most part. Some languages may offer a more filtered version of eval such as python's ast.literal_eval(). In general though it's best to avoid eval and use context specific methods for dealing with user input data.

CVE-2025-48868 is an example of eval being used insecurely.

Shell Parsing

This is a specific security concern for programs which deal with running commands based on dynamic user input variables. An example would be an app that uses ffmpeg to produce thumbnails or re-encode videos. Many of these issue arise if the command is run in a shell environment such as /bin/bash which supports several programming language primitives. In python for example popen can accept a shell=True argument to run it as such.

This can bit tricky to protect against depending on context. It is possible to spawn the command as a simple process to avoid dealing with certain shell primitives being replaced or acting with unintended consequences. That said, you also have to consider if running programs where the arguments are commands to run. Examples of this are sh -c and python -c. You can also associate the process with unprivileged users for most process related functions as well as run the process in a restricted environment.

CVE-2026-40030 is a good example of this being exploited.

Cross Site Scripting (XSS)

This is exploiting string interpolation of the browser. What happens is you have a part of a website, such as a forum post, where users are able to submit their own content. Without any kind of filtering, a user could inject malicious javascript which could redirect a victim to a malware site or view information on the page. Mozilla's developer site has a security article on defending against these sort of attacks which I recommend looking into.

CVE-2024-29184 is an example of an XSS exploit that even shows how you can bypass one of the defenses against it.

SQL Injection

This is a specific exploit against how databases parse queries. It's common enough that there's even an xkcd comic on the topic. This relies on closing out the intended SQL statement early, adding a malicious statement, and then using a comment to ignore the rest and prevent errors from bailing out the call. As an example from the comic:

Robert'); DROP TABLE Students;--

The Robert'); is assuming the query looks something like this:

SELECT email FROM Students WHERE name='$name_here');

So in this case you would get:

SELECT email FROM Students WHERE name='Robert'); DROP TABLE Students;--');

Which effectively turns it into a multi statement query which drops the table entirely. the -- is a comment which tells the SQL parser to ignore the rest of the string. While there are other methods of defense prepared statements with parameterized queries is a popular method of defense. That's because the SQL itself and the input variables are separated out.

CVE-2025-1094 is an example of a SQL injection, though targeting a CLI tool instead of the standard web app exploits you would generally find.

Path Traversal

This is a security issue where code reading string provided directory paths don't filter or restrict properly leaving the ability to access files that aren't intended. Let's say that there's an API endpoint that lets you download a file. In the request you put ../../../etc/passwd. A server which doesn't filter paths would deliver the /etc/passwd file to the user assuming the relative path lead to it. Obtaining the proper amount of directory traversal can be trial and error, though the accuracy can be improved by other means such as information leakage.

Many programming languages work to avoid this by having specific functions which set a path anchor and anything above that will fail. pathlib.PurePath.relative_to in python is also an example of this. Some 3rd party firewalls such as AWS WAF can sometimes have rules to block such attacks.

CVE-2025-68428 is an example of exploiting path traversal.

Wrap Up

If there's one major point to get out of this I would say that in general it's best to not trust user input data. You want to ensure that your risk mitigations assume a malicious actor could abuse user input to attempt an exploit. When you see a string based on user input, consider if there are methods in the language of choice which have context around how you wish to utilize the user data in question. You also want to make sure to have defense in depth. So putting a web application firewall in front of your web servers can help avoid disaster if insecure code manages to slip through. You will want to keep track of what it blocked and insure you're protecting against it on the code side however.

The Road To Kubernetes: How Older Technologies Add Up

Chris White — Mon, 05 Feb 2024 17:16:03 +0000

Virtualization
Resource Isolation
Resource Control
Containers
Docker Joins The Stage
Docker Growth
Introducing Kubernetes
Kubernetes And The Cloud
Component Separation
The Future?

At some point in a DevOps engineer's career they will hear about Kubernetes. It's not a matter of if, but when. With containers being the new hot thing and Kubernetes orchestrating their deployment it's easy to see the reason for its popularity. Even so, Kubernetes is an accumulation of previous technologies. In this article I'd like to discuss the previous way of handling similar application hosting solutions and show how they lead up to a lot of what Kubernetes offers today.

Virtualization

Before containers were popular, virtualization was the way you could split up a single server into multiple "containers". At first they relied on paravirtualization which is what the first EC2 instances used. Virtual Machines would interface to the operating system via some form of middle man which would translate the guest OS's calls to what the host OS would expect. Given the requirement of such a conversion these virtual machines wouldn't be able to keep up with running software straight on the hardware itself. While this wasn't a big issue for some solutions it did mean that adopting it widely where applications cared about performance was a difficult endeavour.

Hardware virtualization would start to gain traction through the usage of various CPU related technologies specifically to improve performance. Technologies such as KVM were made to further improve integration with this new technology. However, there were still issues with some hardware interaction that still made it not match up to bare metal. Another alternative was to simply use the same hardware, but isolate the application itself.

Resource Isolation

Resource isolation primarily started off with isolating how a process could operate. The most popular example of this was chroot (1979). More specialized examples of this included FreeBSD jails (2000) and Solaris Zones (2005) which still kept some of the virtualization aspects. Now chroot, which stands for change root, allowed a process to have their root mount point changed effectively isolating it on a filesystem level.

For these applications to properly run on a Linux system they need access to special paths which provide system information through a filesystem interface. This includes procfs, devpts, and sysfs. These allowed for different mount options for the chroot than the host system they were running on. More simple variants could be done with bind mounts. The underlying application also needed access to its dynamically linked libraries to run. Even static binaries still need access to a c library implementation (ex. glibc) and system calls making setup more difficult if you wanted best performance.

From a security standpoint this setup was beneficial for webservers who only needed access to their respective HTML. CSS, JS, and any dynamic content generation files. If a basic malicious actor obtained access to a local shell, they would only be able to see files inside the chroot. That's not to say that chroot's are some kind of unbreakable fortress. Another issue was that there wasn't a very easy packaging solution which would replicate the ease of use of a virtual machine image. This complexity only increase as more applications are added to a server. Out of the box chroots also don't provide any kind of resource quota mechanism as you would expect from modern day containers.

Resource Control

Several technologies were introduced to the Linux kernel to help address some of these issues. namespaces were introduced into the Linux kernel in 2002 for more fine grained isolation. They allowed for scoping resource availability on a process level. You could have one process see one set of resources, and another process see a different set of resources. Next is quotas on resources which has handled by cgroups, released in 2007. They provide a special filesystem mount to configure settings such as what CPUs a process can utilize, how much memory/cpu it can use, and how much network bandwidth it can use. The final set of controls are capabilities, introduced in 1999. While a foundation for process isolation its original purpose was for restrictions on privileged user accounts. Despite its early introductio, the full set of capabilities available today is something that was an accumulation of additions over several kernel releases. While these are powerful technologies, they are also fairly complicated to manage in a scalable manner. They also had the same issue with chroots in that there wasn't a very easy way to package everything together.

Containers

While Docker helped drive container adoption, it was LXC which provided the base foundation to make the system work. In fact early versions of Docker utilized it as a backend. It provided the packaging of kernel level features such as cgroups and namespaces which were used for container like isolation. Despite it's features LXC still lacked the modern day interface you'd expect to easily deploy containers. There was also a lack of major cloud managed service providers to help with overall adoption.

Docker Joins The Stage

The initial release of Docker was on March 20th 2013. Important to note is that containers are a more of a concept, and Docker is an implementation. What Docker brought to the table was a container image hosting service as well as software to orchestrate the more difficult parts of the container setup process. Docker also provided a very simplistic image definition: the Dockerfile. Compared to what needed to be done for virtual machine style builds Dockerfile builds simplified the process and used an incremental diff patching feature which enabled more modular organization of images. Images hosted on Docker's platform could be utilized in this process allowing for vendor solution tie-in. Having a business backing it also helped with corporate adoption of containers.

Docker Growth

Despite this great set of features Docker still would take some years to become the popular solution it was today. This was attributed to the security and scalability concerns of the original iteration. In 2014 Docker Compose was released as a solution to turning Docker containers into a more "project" like layout. This further helped organization of containers along with the incremental diff functionality. This was further improved via the introduction of Docker Swarm which was released as part of Docker 1.12 in 2016. This allowed a more scalable cluster setup for Docker containers.

Introducing Kubernetes

Kubernetes was first released on September 9, 2014. This release timeline is part of what helped it gain a foothold over Docker Swarm. It was an open source version of an internal Google project. Features of container orchestration were presented in a more modular fashion along with scaling functionality. You can chose how your networking stack works, your load balancing, container runtime, and filesystem interfaces. Availability of an API allowed for more programmatic interactions with orchestration, making it tie in very well with CI/CD solutions. However, the big issue it has is complexity of setup. Putting together a Kubernetes cluster with basic functionality is certainly no easy feat.

Kubernetes And The Cloud

Due to the amount of effort required not to just setup, but also migrate to Kubernetes, original enterprise adaptors would rely on the Google Cloud Kubernetes Engine to abstract the effort. The product was released shortly after Kubernetes itself, with the first release on November 4th 2014. Unfortunately, due to Google Cloud's 2 year gap with AWS's public release this still made Kubernetes adoption on a wider scale more difficult. AWS' first step into the container space was interestingly enough Elastic Container Service or ECS in 2015. While not bound with Kubernetes specifically the ease of deployment made it an interesting competitor for more simplistic container workloads. Google's best alternative to it, Google Cloud Run would not see an available release until 2018.

AWS would then release Elastic Kubernetes Services or EKS in 2018. Azure tagged on slightly afterwards with their own Azure Kubernetes Service or AKS in the same month. With the 3 major cloud providers all having a managed service Kubernetes started to see a large spike in adoption over the years. So much that a majority of DevOps job descriptions now have it as a technology stack component.

Component Separation

Kubernetes on the backend used to utilize docker for much of its container runtime solutions. One of the modular features of Kubernetes is the ability to utilize a Container Runtime Interface or CRI. The problem was that Docker didn't really meet the spec properly and they had to maintain a shim to translate properly. Instead users could utilize the popular containerd or cri-o runtimes. These follow the Open Container Initiative or OCI's guidelines on container formats.

Projects central to the container experience began to join the Cloud Native Computing Foundation. While it supports many popular open source DevOps related projects, much of the governing board and technical oversight committee are corporate oriented. Kubernetes itself also did a migration of their download site from Google to a more open one which would allow for adding additional mirrors.

The Future?

While Kubernetes is in what I consider to be its prime now, like all technologies there's certainly room for a new tech to take its place. Especially true given the complexity of it even when run in a managed environment. A bump in the right direction could certainly be made by having managed services with specific configurations for specific use cases to be available. Centralization of monitoring/security/performance data along with basic administration would also help it substantially.

If you like what you see here I'm currently available for remote full time work. Check my profile for links.

Understanding CVE Reports

Chris White — Sat, 06 Jan 2024 02:18:15 +0000

What Is A CVE
CVE Format
CVE Analysis
- Is The Affected Software Present
- Severity
- Exploit Vector
- Fix Version
- Embargo
- Validation
Having A Reporting Program
Conclusion

CVEs (Common Vulnerabilities and Exposures) are an official collection of recognized security issues. Knowing these is valuable not just for security, but for other teams such as DevOps and development. A major problem is that some people don't know how to handle a CVE properly. This can lead to unnecessary panic and a loss of valuable company time. In this article I'll be going over what a CVE is and how to analyze one.

What Is A CVE

The concept of CVE was created back in 1999 as a way to centralize vulnerability details. While there is a central governing organization, MITRE, actual vulnerability reporting is often handled by other parties. These are known as CVE Number Authorities or CNAs. CNAs span a number of scopes which indicate what CVEs they can report against. For example, Adobe is authorized to publish a CVE for one of it's products, but would not be able to publish a CVE for the Python programming language. Some notable CNA examples:

Adobe
Apple
Python Foundation
Oracle

There are also root CNAs which handle organization of specific scopes of CNAs. Google is one example of a root CNA which covers organization of CVEs related to non Android and Chrome Google products.

CVE Format

CVEs are currently published based off a JSON format specification. Here's an example of an entry for CVE-2023-0038 hosted on the CVE Database in GitHub:

{
    "data_version": "4.0",
    "data_type": "CVE",
    "data_format": "MITRE",
    "CVE_data_meta": {
        "ID": "CVE-2023-0038",
        "ASSIGNER": "",
        "STATE": "PUBLIC"
    },
    "description": {
        "description_data": [
            {
                "lang": "eng",
                "value": "The \"Survey Maker \u2013 Best WordPress Survey Plugin\" plugin for WordPress is vulnerable to Stored Cross-Site Scripting via survey answers in versions up to, and including, 3.1.3 due to insufficient input sanitization and output escaping. This makes it possible for unauthenticated attackers to inject arbitrary web scripts when submitting quizzes that will execute whenever a user accesses the submissions page."
            }
        ]
    },
    "problemtype": {
        "problemtype_data": [
            {
                "description": [
                    {
                        "lang": "eng",
                        "value": "CWE-79 Improper Neutralization of Input During Web Page Generation ('Cross-site Scripting')"
                    }
                ]
            }
        ]
    },
    "affects": {
        "vendor": {
            "vendor_data": [
                {
                    "vendor_name": "ays-pro",
                    "product": {
                        "product_data": [
                            {
                                "product_name": "Survey Maker \u2013 Best WordPress Survey Plugin",
                                "version": {
                                    "version_data": [
                                        {
                                            "version_value": "*",
                                            "version_affected": "="
                                        }
                                    ]
                                }
                            }
                        ]
                    }
                }
            ]
        }
    },
    "references": {
        "reference_data": [
            {
                "url": "https://www.wordfence.com/threat-intel/vulnerabilities/id/a2a58fab-d4a3-4333-8495-e094ed85bb61",
                "refsource": "MISC",
                "name": "https://www.wordfence.com/threat-intel/vulnerabilities/id/a2a58fab-d4a3-4333-8495-e094ed85bb61"
            },
            {
                "url": "https://plugins.trac.wordpress.org/browser/survey-maker/tags/3.1.4/public/partials/class-survey-maker-submissions-summary-shortcode.php?rev=2839688#L311",
                "refsource": "MISC",
                "name": "https://plugins.trac.wordpress.org/browser/survey-maker/tags/3.1.4/public/partials/class-survey-maker-submissions-summary-shortcode.php?rev=2839688#L311"
            }
        ]
    },
    "credits": [
        {
            "lang": "en",
            "value": "Chloe Chamberland"
        }
    ],
    "impact": {
        "cvss": [
            {
                "version": "3.1",
                "vectorString": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:C/C:L/I:L/A:N",
                "baseScore": 7.2,
                "baseSeverity": "HIGH"
            }
        ]
    }
}

The basic breakdown is:

A description of the CVE
Type of vulnerability (cross site scripting)
What is affected
References (such as vendor postings and bug reports)
Credit for report
Overall impact

These are sufficient data points to have a basic idea of if one is affected by a vulnerability.

CVE Analysis

Since we have the information at hand on what a vulnerability is about it's time to analyze it for potential impact. One thing to keep in mind is that each CVE has a set of circumstances to be considered vulnerable. That means specific setups may not be impacted by it, and that upgrades should be handled in an informed manner to reduce unnecessary upgrades which bring unnecessary risk with it.

Is The Affected Software Present

First off is seeing if the software in question is part of your infrastructure. Server software such as nginx are generally easier to tell. In corporate environments there may be a vulnerability scanning solution in place (or something such as dependabot). There could also be some kind of software inventory system in place as well. If you're in a corporate environment and don't have a system in place, I highly recommend looking into it.

Severity

Next to consider is the severity, which is an accumulated score system known as CVSS (Common Vulnerability Scoring System). Calculation of this is based on a number of properties of the exploit, some including:

Is a proof of concept exploit available?
What level of permissions are needed for the exploit?
Is an official fix available?
Is network access required?

and so forth. NIST (National Institute of Standards and Technology) has a calculator with each of the properties explained. The resulting score will indicate how severe the exploit is. For example on the previous wordpress plugin CVE:

    "impact": {
        "cvss": [
            {
                "version": "3.1",
                "vectorString": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:C/C:L/I:L/A:N",
                "baseScore": 7.2,
                "baseSeverity": "HIGH"
            }
        ]
    }

The attack vector is network based, complexity is low, no permissions are required, and there is no user interaction required. This puts it at a higher severity due to this. Even so the resulting impact is fairly low to critical components such as system availability and integrity. Most security teams will want high and above vulnerabilities handled in a very short time frame. Lower severity indicates the ability to pull off the exploit might not be feasible or apply to your particular configuration.

Exploit Vector

Now that severity has been analyzed the next step is to look into how this particular vulnerability is exploited. This is important in cases of highly configurable software such as application servers where features can be enabled/disabled at will. Lets take for example CVE-2022-41741's description text:

NGINX Open Source before versions 1.23.2 and 1.22.1, NGINX Open Source Subscription before versions R2 P1 and R1 P1, and NGINX Plus before versions R27 P1 and R26 P1 have a vulnerability in the module ngx_http_mp4_module that might allow a local attacker to corrupt NGINX worker memory, resulting in its termination or potential other impact using a specially crafted audio or video file. The issue affects only NGINX products that are built with the ngx_http_mp4_module, when the mp4 directive is used in the configuration file. Further, the attack is possible only if an attacker can trigger processing of a specially crafted audio or video file with the module ngx_http_mp4_module.

In this case you'll only be affected if you're using the mp4 module in nginx and the system is allowed to process user input media files. That means if you aren't working with the plugin there is no rush to implement a fix. There also may be environmental aspects to a vulnerability. CVE-2023-5528 for example only affects kubernetes nodes running on Windows systems. Ensuring you understand exploit vectors can help avoid unnecessary prioritization that could put business value add tasks on hold.

Fix Version

This is one of the areas where CVE scanning may show its true colors. Taking a CVE at face value, you might think you simply update to the latest version and then it's done. It's not that simple however. The version you're currently on and the version to upgrade two may contain changes that are not related to the vulnerability at all. This means introducing additional risk into systems in case another non-security bug is present. Due to this it's not uncommon for vendors to implement backports. This is where they take the patch and make it work for a previous version of the package. This helps reduce the introduction of risk into systems. For example, if we look at CVE-2023-46724:

{
    "product_name": "squid",
    "version": {
        "version_data": [
            {
                "version_affected": "=",
                "version_value": ">= 3.3.0.1, < 6.4"
            }
        ]
    }
}

The affected versions are greater than 3.3.0.1 and less than 6.4. However, if we look at the RedHat security advisory you'll notice that the fixed package is "squid:4" or more specifically squid-4.15-7.module+el8.9.0+20975+25f17541.5.x86_64.rpm. Due to RedHat Linux being targeted to slower moving enterprise customers, it's within their business interest to backport instead of updating to the mentioned version 6.4 in the original report. This is why when analyzing a CVE you need to understand where your source of the package is, for example:

Your distribution
Part of a container or virtual machine image
Part of a language's package manager
Manual installation via automation such as AWS Systems Manager or Ansible

It is important to note that you may not always have the luxury of a backport available. In such cases you can attempt to do it yourself, or just deal with the added risk of unrelated changes (going over the changelog of a package is a good idea).

Embargo

This is a special case for CVEs which have an extremely large number of affected systems and require coordinated effort to push a release out. OpenSSL is where you'd be likely to see an embargo occur (this happened somewhat recently with CVE-2022-3786). This gives time for vendors to upgrade systems / packages (such as Microsoft for example) and allows companies to ensure proper staffing to deal with the issue. The primary purpose of all of this is to prevent the possibility of exploits out in the wild causing a war room situation. That's not to say an exploit won't show up, but the less the chance the better.

Validation

Now assuming you are affected it's time to take action. I will note that some extremely serious vulnerabilities (such as embargo'ed ones) you may not have the luxury of full validation due to the time sensitive nature of it. How you achieve a package update will depend on your particular setup and the source of your packages. Depending on the impact of updating packages it may require a scheduled maintenance window. Before deployment though it's recommended to do some validation. If the report has a proof of concept it's possible to use this to validate that your upgrade will work. You should only use this to validate if the PoC has no destructive operations such as deleting files or DDoS-ing a server. I recommend doing an upgrade and run of the PoC ON A NON PRODUCTION DEV/TEST SYSTEM. If you're hosting on a cloud provider such as AWS you might need permission ahead of time depending on how the PoC operates.

You'll also want to do system integrity validation with the new version. This is where you do the upgrade, then run your test suites (or cry because you don't have any) to ensure nothing will break. A broken server can sometimes be seen as worse as a vulnerable one. In some cases you may need to report back to the original bug report where the issue was found. Once everything looks good you can look towards moving it to productions. Blue/Green deployment is another method of helping ensure everything is good by doing a test deployment with validation to ensure critical applications are available.

Having A Reporting Program

While I've covered how to analyze CVEs there are cases where software you work on may be the target of a vulnerability. The first step is having a dedicated channel for reporting potential security vulnerabilities. It should be highly monitored and if possible initial response made within the same business day. Always keep in communication with the reporter. Failure to do so mean they could potentially release their vulnerability findings (with a potential exploit) before you have a time to fix it. If you expect that security reports may be somewhat more common it might be a good idea to become a CNA and report CVEs on your own.

Some organizations even go so far as to provide financial incentive to report security issues through bug bounty programs. The Department Of Homeland Security has a bug bounty program for example. In some cases you may see amounts anywhere from $10,000 - $20,000 depending on the impact of the security vulnerability. Whether or not an organization needs a bug bounty program very much will depend on how high value of a target it is (a mom and pop shop's online store is probably not a good case for one).

Conclusion

I hope this article has provided insight on how CVEs are formatted, and what information you can get out of them. Being able to understand the potential impact of a CVE can go a long way in ensuring the relevant fixes are properly prioritized, and it's not putting crucial project work at risk unnecessarily. I highly recommend understanding how CVEs work even if you're not in a security specific role. It can really help if you need to implement the fixes yourself.

Different Levels of Project Documentation

Chris White — Fri, 29 Dec 2023 22:46:56 +0000

Naming Convention Documentation
Typing Documentation
Comment Documentation
Project README
Internals Documentation
Process Documentation
General Usage Documentation
API Documentation
Advanced Documentation
External Documentation
Hosting Documentation
Conclusion

Documentation is extremely valuable for both open source and internal company projects. With that in mind the question arises on what exactly needs to be documented. While code docs generated from inline formats (ex. python doc strings) may be what is considered to be documentation for developers, there's more to it than that. In this article I'll be breaking down different kinds of documentation and what gives them value to their target audience. Layout wise I'll start at the lowest level and move upwards.

Naming Convention Documentation

First off is the naming of variables and functions which drive the code. A common mistake for newer developers is having code with a lot of inline comments. Much of these verbose comments could be addressed by simply having a naming convention which provides clarity on what the purpose of the function or variable is. As an example:

isAddressPOBox()
Address.isPOBox()
ec2_boto_client

The first and second example shows how naming can change between context. Meanwhile ec2_boto_client could be useful if you're dealing with multiple boto clients connecting to the AWS API. Something to note here is that isAddressPOBox() could technically just be isPOBox but I find that the verbose version also gives context on the input. I would say that you want to get into the habit of doing this for even personal pet projects. It's also very valuable in team environments to make code easier to read (and potentially faster PRs).

Typing Documentation

Typing is useful for giving hints on function input and output. While this is default for statically typed languages, some dynamic languages such as python may support type hinting features for this particular purpose:

def average_numbers(numbers: list[int]) -> float:

In this example I know I need to provide a list of integers as input and will obtain a float as output. You will want to be careful on how you organize custom types. A lack of organization can lead to a lot of back and forth on source code reading. Another nice benefit of typing is that it can be used by many IDEs to enhance linting features by seeing if inputs and outputs match the requested types. This is useful for cases where your project is a dependency library as well as working on team environments.

Comment Documentation

Documentation through code comments can be useful in cases where code may rely on complex algorithms or when something is done in an unusual manner. I find most of the code I comment on tends to be related to cryptographic hashing:

).sign(self._key_object, hashes.SHA256())
# SHA256 was chosen here instead of SHA512 as a compromise between decrypt performance and
# hash security

In this example I'm explaining the reasoning why I'm using SHA256 to sign a cert when the more powerful SHA512 hash exists. TODO comments are a specialized form of comment documentation which allows developers to keep track of code improvements quickly, which can be transitioned to tickets/issues/tasks later on. I will warn on being careful on your ratio of comments to code. At a certain point the actual code becomes difficult to follow and could be a sign that your naming convention/typing might have issues.

## Application Interface Documentation

This style of documentation is often what developers consider to be "code documentation". Users may reference it for libraries to figure out what functions need to be called. Developers may use it to understand the code base. Such documentation is often found after a function/method signature to document inputs, outputs, and a basic summary of what the code does:

 def average_numbers(numbers: list[int]) -> float:
    """Average a list of numbers

    :param numbers: The list of numbers to average
    :type numbers: list[int]

    :return: The average of the numbers
    :rtype: float
    """
    return np.average(numbers)

Important to note is that the format of application interface documentation will vary depending on the programming language and documentation solution. If you find that files become too verbose due to application interface documentation it might be a sign of functions trying to do too much or a need for separation refactoring.

Project README

I would consider this the starting point of figuring out basic information for a project. This includes items such as:

Why the project was created
Status of the project
Project requirements
How to install the project (or a link to an install guide for more complex projects)
Basic usage / getting started
Security policy
Links to further documentation
Contact information (filing bug reports, etc.)

A well laid out README can often be a great way to get project adoption as it brings an assumption that the rest of the project is well documented and users will find information they need easily.

Internals Documentation

This somewhat ties in with application interface documentation. It's general used to supplement such documentation and primarily geared towards developers, power users, and potential contributors to a project. A few examples of internals documentation:

In general this type of documentation tends to be useful at larger scale products with wide user adoption. Understanding internals of a project can also help in finding potential performance optimizations or conditions that might cause bugs to occur.

Process Documentation

This is used to describe various processes a project might have in dealing with the codebase. Some examples of this include:

How to contribute to a project
Setting up a development environment
Project code of conduct (in particular enforcement of it)
Bug reporting
Security reporting guidelines (different from basic bug reporting as it tends to be done in a more private manner to avoid early exploits in the wild)
Release process

It's essentially how users might interact with a project versus how to use the actual project itself. This category of documentation is primarily oriented towards projects with a high contribution rate.

General Usage Documentation

General usage documentation is geared towards end users of the project. This may replace parts of the README documentation if certain usage details require a substantial explanation. General usage documentation often touches on the following components:

Downloading the project (source code/package/installer)
Installing the project
Configuring the project
Validating the project (generally in the form of a quick start guide)

Application interface documentation may be included here as well if the project in question is primarily used as a dependency. Smaller projects may combine download, installation, and configuration steps. Larger projects may need them broken out to handle environment variations such as operating systems, package managers, service management, database engines, etc.

API Documentation

Often used for cases where a project exposes a REST or other type of API service. Open API is a popular method of documenting such API services. It can also be used along side tools such as Swagger Codegen to produce boilerplate code for API interaction / testing purposes. There may also be support files for popular API testing tools such as Postman or Insomnia. This makes it easier at a glance to see what data is coming back from a call so the user knows how to handle parsing the data.

Advanced Documentation

General usage documentation works well for covering the standard user needs for the project. Advanced documentation enhances this by showing more advanced setups which are useful but not what you would expect from an out of the box setup. Such documentation could touch on manual configurations that are normally set to developer recommended settings to cover a majority of use cases. Finally there are cases where subject matter expertise is required such as BGB network integration with Kubernetes networking providers. Such documentation is generally recommended for more active projects where the developer is more aware of project use cases.

External Documentation

This may also be considered "community documentation". Someone writes a blog post on a project, maybe a key note video is posted, essentially anything that isn't directly hosted on core project infrastructure. It may also point to community resources such such as forums, slack channels, and social media sites. One word of caution is that I recommend avoiding slack exclusive documentation for a project as there is a barrier of entry in accessing the information (not to mention separation of concerns).

Hosting Documentation

Once you have all the documentation worked out a place to host it will be necessary. Some documentation generation may have ties in with specific hosting sites. Read The Docs' support for Sphinx and other documentation tools is one example. GitHub pages can be useful for GitHub hosted projects as it integrates well with GitHub Actions CI/CD deployments.

If you want to self host and don't mind paying a bit a combination of Route53 (unless you host DNS elsewhere), S3, and CloudFront on AWS can provide a pretty reliable and cost effective site hosting. It's also a great solution if you're working on an internal company project with architecture hosted on AWS. It's also nice in compliance environments since you can implement strict access controls when necessary which can integrate with company SSO and other IAM components.

Wikis are another solution if your contributors are more comfortable with them. It also helps prevent the issue of "documentation updates caused a big CI run" that can catch some projects off guard. For GitHub users there is a fairly minimal GitHub Wiki feature for projects that support it. This also works on the enterprise versions if you're doing an internally hosted project.

Conclusion

As someone who has been a proponent of documentation for much of my career, I hope this provides some insight on the depth of documentation you can have for a project. I will say that some types of documentation tend to show their true value at certain project growth stages. Trying to do advanced usage documentation is not ideal when your project is starting out and you don't have enough usage information. With that said I recommend looking over each documentation type and evaluating if it would help with project adoption / contributions.

Implementing Quality Checks In Your Git Workflow With Hooks and pre-commit

Chris White — Wed, 13 Dec 2023 15:45:40 +0000

Hooks and Webhooks
Git Hook Basics
Practical Git Hook Usage
Git Hook Automation With pre-commit
Working With Contributed pre-commit Code
File Filtering
Conclusion

Git is a powerful tool to enable developers to organize and share their code. One of the more interesting features is hooks. This allows for execution of scripts at certain points in the git workflow. In this article I'll be showcasing one of the simple git workflow events: pre-commit. I'll also be introducing a tool to help with automation of it, the aptly named pre-commit. Note that the code used for this is the final step of my python beginners series and can be found in a code repository in GitHub. Simply click on the green "Code" button and select "Download ZIP". Then extract it into your preferred directory of choice.

Hooks and Webhooks

To clear up any potential confusion, Git hooks and GitHub webhooks are different entities (though GitHub webhooks most likely is powered by git hooks). Git hooks are specific to the git software package. GitHub webhooks are a feature of the GitHub platform that pushes JSON payloads based on various GitHub related events. The same goes for similar source control service sites such as GitLab

Git Hook Basics

A git hook is essentially a script that is executed at a certain point in the git workflow. The basic rules of hooks are:

Set as executable
Strip the extension (with the exception of certain windows extensions such as .exe)
Located in a .git/hooks directory under the repository parent directory

To see how this looks download the code mentioned in the introduction paragraph and make sure there's no .git directory in it (remove it if there is). Then run git init to initialize the repository:

$ git init
$ git branch -m main

The second command ensures the default branch is main, which is fairly common for new repositories on major code hosting sites. After the git repository has been initialized it's time to look at the contents of .git/hooks:

$ cd .git/hooks
$ ls -1
applypatch-msg.sample
commit-msg.sample
fsmonitor-watchman.sample
post-update.sample
pre-applypatch.sample
pre-commit.sample
pre-merge-commit.sample
prepare-commit-msg.sample
pre-push.sample
pre-rebase.sample
pre-receive.sample
push-to-checkout.sample
update.sample

Each of these has examples. To make them work we'll need to make a copy with .sample removed and ensure it's executable. Let's take a look at the pre-commit.sample contents:

#!/bin/sh
#
# An example hook script to verify what is about to be committed.
# Called by "git commit" with no arguments.  The hook should
# exit with non-zero status after issuing an appropriate message if
# it wants to stop the commit.
#
# To enable this hook, rename this file to "pre-commit".

if git rev-parse --verify HEAD >/dev/null 2>&1
then
        against=HEAD
else
        # Initial commit: diff against an empty tree object
        against=$(git hash-object -t tree /dev/null)
fi

# If you want to allow non-ASCII filenames set this variable to true.
allownonascii=$(git config --type=bool hooks.allownonascii)

# Redirect output to stderr.
exec 1>&2

# Cross platform projects tend to avoid non-ASCII filenames; prevent
# them from being added to the repository. We exploit the fact that the
# printable range starts at the space character and ends with tilde.
if [ "$allownonascii" != "true" ] &&
        # Note that the use of brackets around a tr range is ok here, (it's
        # even required, for portability to Solaris 10's /usr/bin/tr), since
        # the square bracket bytes happen to fall in the designated range.
        test $(git diff --cached --name-only --diff-filter=A -z $against |
          LC_ALL=C tr -d '[ -~]\0' | wc -c) != 0
then
        cat <<\EOF
Error: Attempt to add a non-ASCII file name.

This can cause problems if you want to work with people on other platforms.

To be portable it is advisable to rename the file.

If you know what you are doing you can disable this check using:

  git config hooks.allownonascii true
EOF
        exit 1
fi

# If there are whitespace errors, print the offending file names and fail.
exec git diff-index --check --cached $against --

So this will do a few basic checks against file names and whitespace issues. As one of the comments mentions To enable this hook, rename this file to "pre-commit". We can then just run git commit afterwards to run it:

$ cp pre-commit.sample pre-commit
$ cd ../../
$ git commit

At this point nothing happened as nothing was added. I'll go ahead and add a file with a Japanese name to see what happens:

$ touch テスト
$ git add テスト
$ git commit
$ git commit
Error: Attempt to add a non-ASCII file name.

This can cause problems if you want to work with people on other platforms.

To be portable it is advisable to rename the file.

If you know what you are doing you can disable this check using:

  git config hooks.allownonascii true

As you can see our hook is working and is preventing the non ASCII named file from being committed. I'll go ahead and cleanup the test:

$ git reset テスト
$ rm テスト

Practical Git Hook Usage

So we've seen a very basic example of how hooks work. Now the more practical example would be to use it to run tests on code before committing. Note that I don't recommend this if your project is in the prototyping phase where you may be doing many commits and may not have tests setup due to the frequency of code architecture changes. In this case the project in question has tox setup which runs various python linting and tests. Before running the examples you'll need to ensure pdm is installed and running under python 3.11. Instructions for this can be found in my pdm tutorial, or you can install it on your own. Once installed we'll make sure the necessary packages are available for tox to work with:

$ pdm install

Then I'll edit the .git/hooks/pre-commit file to contain the following:

#!/bin/sh
pdm run tox

if [ $? -ne 0 ]; then
  echo "tox checks failed" >&2
  exit 1
fi

Now when git commit runs:

$ git commit
lint: install_deps> pdm sync --no-self --group testing --group lint
lint: commands[0]> flake8

tox is run to initiate all the checks. As is the code should be in working condition so you'll see a commit screen after the tox run. Go ahead and close it out without putting a message to abort the comment. When there's an issue with something (I went ahead and commented out one of the imports):

$ git commit
lint: install_deps> pdm sync --no-self --group testing --group lint
lint: commands[0]> flake8
<snip>
  lint: FAIL code 1 (4.34=setup[3.69]+cmd[0.65] seconds)
  test: FAIL code 1 (11.80=setup[10.09]+cmd[1.71] seconds)
  docs: OK (14.44=setup[11.79]+cmd[2.65] seconds)
  evaluation failed :( (30.79 seconds)
tox checks failed

A message at the end shows that tox has failed to run and no commit screen is displayed.

Git Hook Automation With pre-commit

Given how useful checking code quality is before committing, there's actually a framework around git hooks called pre-commit. It's somewhat like a mini-GitHub Actions that can be used to run various commands during the pre-commit phase. This is where you normally want most CI/CD like checks. Before continuing you'll want to remove the existing hook or you'll end up having duplicated tool runs. We'll also go ahead and add all the files in the repo so there's something to check:

$ rm .git/hooks/pre-commit
$ git add .

Given how useful pre-commit is across projects I generally recommend installing via pip install --user, making it part of a tooling virtual environment, or using pipx:

$ pip install --user pre-commit
$ pipx install pre-commit

Next we'll need to create a YAML configuration file in the root of our repository called .pre-commit-config.yaml. To make things simple you can bootstrap a basic one via:

$ pre-commit sample-config > .pre-commit-config.yaml

As is the file looks something like this:

# See https://pre-commit.com for more information
# See https://pre-commit.com/hooks.html for more hooks
repos:
-   repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v3.2.0
    hooks:
    -   id: trailing-whitespace
    -   id: end-of-file-fixer
    -   id: check-yaml
    -   id: check-added-large-files

This already has some great entries for basic git related checks. Now we'll add a new entry to cover tox checking:

# See https://pre-commit.com for more information
# See https://pre-commit.com/hooks.html for more hooks
repos:
-   repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v3.2.0
    hooks:
    -   id: trailing-whitespace
    -   id: end-of-file-fixer
    -   id: check-yaml
    -   id: check-added-large-files
-   repo: local
    hooks:
    -   id: tox check
        name: tox-validation
        entry: pdm run tox
        language: system
        types: [python]
        pass_filenames: false

repo: local tells pre-commit that what I'm doing is not a managed action by code someone else wrote. It's completely custom code written by me. The code itself will run pdm run tox in the system environment when it finds python files have been modified. This is one of the more powerful features of pre-commit over the standard git hook. There's more flexibility on when certain commands get run. You don't need to check tests if only the README file was updated. Now to actually have this integrated with our workflow we'll need to run:

$ pre-commit install
pre-commit installed at .git/hooks/pre-commit

The generated file is simply an entry point to the actual pre-commit program which handles the processing of what we want to do:

#!/usr/bin/env bash
# File generated by pre-commit: https://pre-commit.com
# ID: 138fd403232d2ddd5efb44317e38bf03

# start templated
INSTALL_PYTHON=/home/johndoe/.pyenv/versions/py311/bin/python3.11
ARGS=(hook-impl --config=.pre-commit-config.yaml --hook-type=pre-commit)
# end templated

HERE="$(cd "$(dirname "$0")" && pwd)"
ARGS+=(--hook-dir "$HERE" -- "$@")

if [ -x "$INSTALL_PYTHON" ]; then
    exec "$INSTALL_PYTHON" -mpre_commit "${ARGS[@]}"
elif command -v pre-commit > /dev/null; then
    exec pre-commit "${ARGS[@]}"
else
    echo '`pre-commit` not found.  Did you forget to activate your virtualenv?' 1>&2
    exit 1
fi

Now after this I'll add the pre-commit YAML as there's a sanity check to ensure it's part of the repository:

$ git add .pre-commit-config.yaml

This is because the end goal is that anyone who downloads the code can install your pre-commit setup themselves and be able to run the appropriate tests. Now looking at the result:

$ git commit
[INFO] Installing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
Trim Trailing Whitespace.................................................Failed
- hook id: trailing-whitespace
- exit code: 1
- files were modified by this hook

Fixing README.md

Fix End of Files.........................................................Passed
Check Yaml...............................................................Passed
Check for added large files..............................................Passed
tox-validation...........................................................Failed

So as the file I purposely broke hasn't been fixed the tox-validation stage has failed. It also found an issue with trailing whitespace in my README.md and even fixed it for me. I'll go ahead and add the updated README and fix the commented out import so things are working again:

$ git add README.md
$ vim src/my_pdm_project_cwprogram_test/mymath.py
$ git add src/my_pdm_project_cwprogram_test/mymath.py
$ git commit
Trim Trailing Whitespace.................................................Passed
Fix End of Files.........................................................Passed
Check Yaml...............................................................Passed
Check for added large files..............................................Passed
tox-validation...........................................................Passed

Now that everything has passed I'm given the ability to commit. The output is also much more user friendly and mimics what you'd expect from a CI/CD or test suite run.

Working With Contributed pre-commit Code

In the configuration file you might have noticed:

repo: https://github.com/pre-commit/pre-commit-hooks

The basic functionality of this works similar to a GitHub repo with GitHub Actions code. You can even see the code for end_of_file_fixer as an example:

def fix_file(file_obj: IO[bytes]) -> int:
    # Test for newline at end of file
    # Empty files will throw IOError here
    try:
        file_obj.seek(-1, os.SEEK_END)
    except OSError:
        return 0
    last_character = file_obj.read(1)
    # last_character will be '' for an empty file
    if last_character not in {b'\n', b'\r'} and last_character != b'':
        # Needs this seek for windows, otherwise IOError
        file_obj.seek(0, os.SEEK_END)
        file_obj.write(b'\n')
        return 1

In fact there's actually a fairly sizeable amount of these listed on the pre-commit website. In fact another one is provided by the PDM project to make sure pdm.lock is up to date. I'll go ahead and add this in:

# See https://pre-commit.com for more information
# See https://pre-commit.com/hooks.html for more hooks
repos:
-   repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v3.2.0
    hooks:
    -   id: trailing-whitespace
    -   id: end-of-file-fixer
    -   id: check-yaml
    -   id: check-added-large-files
-   repo: local
    hooks:
    -   id: tox check
        name: tox-validation
        entry: pdm run tox
        language: system
        types: [python]
        pass_filenames: false
-   repo: https://github.com/pdm-project/pdm
    rev: 2.10.4 # a PDM release exposing the hook
    hooks:
    -   id: pdm-lock-check

Now I'll manually edit one of my dependencies in pyproject.toml and check the result:

dependencies = [
    "numpy>=1.25.1",
    "requests>=2.31.0",
]

$ git add pyproject.tmol .pre-commit-config.yaml
$ git commit
[INFO] Initializing environment for https://github.com/pdm-project/pdm.
[INFO] Installing environment for https://github.com/pdm-project/pdm.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
Trim Trailing Whitespace.................................................Passed
Fix End of Files.........................................................Passed
Check Yaml...............................................................Passed
Check for added large files..............................................Passed
tox-validation...........................................................Passed
pdm-lock-check...........................................................Failed
- hook id: pdm-lock-check
- exit code: 1

Lock file hash doesn't match pyproject.toml, packages may be outdated

In this case it noticed that my numpy dependency has changed. I'll go ahead and revert this and see what happens:

$ vim pyproject.toml
$ git add pyproject.toml
$ git commit
Trim Trailing Whitespace.................................................Passed
Fix End of Files.........................................................Passed
Check Yaml...............................................................Passed
Check for added large files..............................................Passed
tox-validation...........................................................Passed
pdm-lock-check...........................................................Passed

Now everything is looking good and I'm allowed to commit again. Trying to check if the pdm lock file was synced would have taken a substantial amount of time via a normal git hook, primarily due to not having context on the pdm code base. Instead I can use this hook provided by the developers in less than ten minutes to do it instead.

File Filtering

To showcase this best we'll want to commit what we have now so there's better control over what files get added:

$ git commit -m "Initial Commit"

Now since tox can run specific stages we can use this to break out tasks based on what's actually been committed. Considering when we'd want things to run:

All files should have the basic large files, trailing whitespace, and end of files check
README.md should have a markdown linter run against it
pyproject.toml should have a toml linter run against it
Linting should be run if files in src or tests are modified
Tests should be run if files in src or tests are modified
Documentation building should be run if files in src (for automodule generation) or docs are modified
Everything should run if pyproject.toml is updated as it controls settings and manages dependencies

So let's see what a solution like this would look like:

# See https://pre-commit.com for more information
# See https://pre-commit.com/hooks.html for more hooks
repos:
-   repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v3.2.0
    hooks:
    -   id: trailing-whitespace
    -   id: end-of-file-fixer
    -   id: check-yaml
    -   id: check-toml
    -   id: check-added-large-files
-   repo: local
    hooks:
    -   id: tox lint
        name: tox-validation
        entry: pdm run tox -e test,lint
        language: system
        files: ^src\/.+py$|pyproject.toml|^tests\/.+py$
        types_or: [python, toml]
        pass_filenames: false
    -   id: tox docs
        name: tox-docs
        language: system
        entry: pdm run tox -e docs
        types_or: [python, rst, toml]
        files: ^src\/.+py$|pyproject.toml|^docs\/
        pass_filenames: false
-   repo: https://github.com/pdm-project/pdm
    rev: 2.10.4 # a PDM release exposing the hook
    hooks:
    -   id: pdm-lock-check
-   repo: https://github.com/jumanjihouse/pre-commit-hooks
    rev: 3.0.0
    hooks:
    -   id: markdownlint

First off we have check-toml for linting our pyproject.toml file. markdownlint is taken from jumanjihouse/pre-commit-hooks. Note that check-* and the markdown linter generally has code to check if appropriate files were added so there's no need to add filters. Another thing to keep in mind is that it's recommended to pin rev against a specific revision (usually a tag) versus something like master or main. This will reduce "unexpected" surprises due to code changes. Here we see the actual filtering at work:

    -   id: tox lint
        name: tox-validation
        entry: pdm run tox -e test,lint
        language: system
        files: ^src\/.+py$|pyproject.toml|^tests\/.+py$
        types_or: [python, toml]

So files: ^src\/.+py$|pyproject.toml|^tests\/.+py$ is a regular expression to show what files we're interested in. In this case it's files under src/ and tests/ as well as pyproject.toml. types_or (requires 2.9.0 or later pre-commit) also ensures we're only looking at python or toml files. If you're wondering what to put in for types_or the identify-cli tool will let you know appropriate values that can be used:

$ identify-cli docs/source/index.rst
["file", "non-executable", "rst", "text"]

file is the most generic type. You can also be more specific with rst for example. types can be used if you want something that works off AND comparison instead:

        types: [python, executable]

This will run if a file is a python file and executable as well. Now that we've seen how comparisons work, it's time to put this into practice. I'll add our updated pre-commit YAML, modify README.md, and then run git commit:

$ vim README.md #changes here
$ git add .pre-commit-config.yaml README.md
$ git commit
[INFO] Initializing environment for https://github.com/jumanjihouse/pre-commit-hooks.
[INFO] Installing environment for https://github.com/jumanjihouse/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
Trim Trailing Whitespace.................................................Passed
Fix End of Files.........................................................Passed
Check Yaml...............................................................Passed
Check Toml...........................................(no files to check)Skipped
Check for added large files..............................................Passed
tox-validation.......................................(no files to check)Skipped
tox-docs.............................................(no files to check)Skipped
pdm-lock-check.......................................(no files to check)Skipped
Check markdown files.....................................................Passed

Given that no toml or appropriate python files were modified unnecessary checks are skipped according to our setup. The README.md file does have markdownlint run against it and has passed the checks. I'll go ahead and commit the file with the message "Updated README.md title". Now it's time to see what happens when we make changes to docs/:

$ vim docs/source/index.rst
$ git add docs/source/index.rst
$ git commit
Trim Trailing Whitespace.................................................Passed
Fix End of Files.........................................................Passed
Check Yaml...............................................................Passed
Check Toml...........................................(no files to check)Skipped
Check for added large files..............................................Passed
tox-validation.......................................(no files to check)Skipped
tox-docs.................................................................Passed

In this case tox-docs is run but since no python files are present neither linting nor tests were run. Now I'll revert the docs change and modify one of the tests. This should kick off the linting/tests phase but not do anything with the docs building:

$ git reset docs/source/index.rst
$ git checkout docs/source/index.rst
$ vim tests/test_mymath.py
$ git add tests/test_mymath.py
$ git commit
Trim Trailing Whitespace.................................................Passed
Fix End of Files.........................................................Passed
Check Yaml...............................................................Passed
Check Toml...........................................(no files to check)Skipped
Check for added large files..............................................Passed
tox-validation...........................................................Passed
tox-docs.............................................(no files to check)Skipped
pdm-lock-check.......................................(no files to check)Skipped
Check markdown files.................................(no files to check)Skipped

Finally we'll make an update to pyproject.toml which should run everything (I'll also reset the state of the test file so there are no false positives):

$ git reset tests/test_mymath.py
$ git checkout tests/test_mymath.py
$ vim pyproject.toml
$ git add .pre-commit-config.yaml pyproject.toml
$ git commit
Trim Trailing Whitespace.................................................Passed
Fix End of Files.........................................................Passed
Check Yaml...............................................................Passed
Check Toml...............................................................Passed
Check for added large files..............................................Passed
tox-validation...........................................................Passed
tox-docs.................................................................Passed
pdm-lock-check...........................................................Passed
Check markdown files.................................(no files to check)Skipped

Everything has been run. I will say that technically it wasn't necessary to run this as I only made a description update and nothing was changed in the actual dependencies or tool configuration. That means nothing was changed that would require linting/tests/documentation updates. Even so, trivial pyproject.toml changes will generally be very rare after the project is flushed out and you should only really be updating for dependency or tool configuration changes from there.

Conclusion

pre-commit is definitely one of the tools I wish I knew about sooner in my development career. It's a great way to avoid having people frown at your PRs for having 3 different lint fix only commits. Given that it is another roadblock to pushing commits you'll want to make sure your linting passes on the entire project via pre-commit run -a. Otherwise your coworkers will most definitely find a way to bypass it.

If you like what you see I'm am also available for hire. Those interested can find out more info in my dev.to profile.

Publishing Python Packages To PyPI With Twine

Chris White — Wed, 06 Dec 2023 21:21:06 +0000

PyPI Background
PyPI Account Creation
Package Refactor
Twine Upload
Adding Metadata
- Classifiers
Conclusion

In the last installment we looked at using tox to centralize our development tool workflow. We now have tests, n centralized way to check our code for issues, and documentation generation all in a single command. Now that our project is solid enough we need to look into releasing it to the general public. In this article we'll look at twine and using it to upload to the official PyPI servers.

PyPI Background

The first step to uploading code is having someplace to upload it to. Python's official package repository is PyPI. This is similar to NPM if you're familiar with NodeJS. It's a centralized location for both uploading and downloading python packages. Packages here are meant to handle everything from common to complex tasks so you don't have to write it yourself. It does mean, however, that you have to be careful about what you install. Malicious packages has been a known issue to not just PyPI, but many other popular package repositories as well. Packages without tests to show confidence that it works or documentation to show how you use it may cost you more time than you save. Finally, you don't want to be relying on code you expect to evolve over time if it's not longer being maintained.

PyPI Account Creation

Before we upload anything we'll need a PyPI account. In this case I actually recommend doing so on their test instance. This allows you to test your package uploads and installation without causing impact to the main PyPI servers. The sign in form is simple with the usual name, address, username, password, and captcha verification:

After completing the signup take note of the username as it will be required later. There's also an email verification step you'll need to complete to ensure you actually own the email address. After this you'll want to setup two factor authentication or 2FA for short. This means you login with two "factors": something you remember and something you own. The PyPI help page has information on setting this up for one of two methods:

Software based, such as Google Authenticator
Hardware based, such as a YubiKey

While YubiKeys are a bit on the pricey side the ability to authenticate by simply plugging in a device (and maybe taping a part of it) is very easy to do. Software solutions that work off of a smart phone can become an issue if your smart phone is misplaced or stolen. Once this is setup you'll want to go to your manage account page and there is an API token option towards the bottom:

Now tokens are the preferred way to authenticate against the system as if they are compromised you can simply generate a new one. If you use your account username and password instead and it's compromised getting your account back can be tedious. Clicking on Add API Token will lead to a screen that you can fill out like this:

On the test instance with your new account "Scope" only has one option so the warning can be ignored. Be sure to copy down the value as it will disappear afterwards and you'll have to regenerate it later if you forget. Now that our authentication details are available it's time to setup the upload process.

Package Refactor

Now up until now we've been using the name my_pdm_project to represent our package. Unfortunately the issue with this method is that PyPI allows one unique package name globally. This means if someone were to upload my_pdm_project you wouldn't be able to upload your package code as-is to PyPI. To work around this we're going to rename the project to my_pdm_project_[pypiusername] where [pypiusername] is replaced with the username you chose during the PyPI registration process. In this example I'll use my_pdm_project_cwprogram_test (be sure to not actually use this but the version with your username instead). Please note that this is only this one specific case for general tutorial usage and you normally wouldn't put your username in the package name like that. So let's go ahead and make the changes. The first thing you'll need to do is go to the src directory and rename your my_pdm_project to the new one:



$ cd src
$ mv my_pdm_project my_pdm_project_cwprogram_test

pyproject.toml needs to be updated to reflect the new package name. Also since we're here we're going to update the version to 1.0.0 as this is considered our finalized public release:



[project]
name = "my-pdm-project-cwprogram-test"
version = "1.0.0"

Test suites need to be updated to point to the new package location in tests/test_mymath.py:



from my_pdm_project_cwprogram_test.mymath import (
    add_numbers,
    average_numbers,

tox.ini needs an update so that code coverage is using the proper module name:



[testenv:test]
groups = testing
commands =
  pytest --cov=my_pdm_project_cwprogram_test

Sphinx will need updates in package name and version via docs/source/config.py:



project = 'my-pdm-project-cwprogram-test'
copyright = '2023, Chris White'
author = 'Chris White'

version = '1.0.0'
release = '1.0'

Finally we'll update the module name for our automatic api documentation in docs/source/mymath.rst:



Mymath Module Documentation
===========================
.. automodule:: my_pdm_project_cwprogram_test.mymath
    :members:

To make sure we didn't miss anything:



> pdm run tox
<snip>
  lint: OK (4.59=setup[0.84]+cmd[0.56,3.19] seconds)
  test: OK (3.59=setup[3.05]+cmd[0.55] seconds)
  docs: OK (3.55=setup[2.86]+cmd[0.69] seconds)
  congratulations :) (11.81 seconds)

Now that everything looks good it's time to begin the upload process (I also hope this is a good lesson on why you should stick with a project name once you've decided on it).

Twine Upload

Up until now I've recommended adding tools to our dev group in the PDM project. However twine is something where you want the authentication centralized and usable by all python environments. This fits the description of something that should be installed with pipx instead:

> pipx install twine

Now we need to configure authentication for twine to upload packages. For simplicity's sake I will be showing how to do this via a simple text file. Once you have gained more experience as a developer I highly recommend switching to keyring as a more secure method (the sooner the better) so your password isn't in a plain text file (I avoided doing it here as keyring can be overwhelming for new developers). To get started you'll need to create a .pypirc file in your home directory. For Linux and OSX you should be able to shortcut it as ~/.pypirc for Windows it will be in %HOME%\.pypirc in command prompt and $env:HOME\.pypirc in Powershell. Here is how the contents should look:



[distutils]
index-servers =
    testpypi

[testpypi]
repository = https://test.pypi.org/legacy/
username = __token__
password = pypi-[rest_of_token]

__token__ as the username is a special indicator that we're using an API token instead of a regular username and password. password is the API token value you obtained when generating it, which starts with pypi-. Now that our authentication is in place it's time to test uploading to PyPI. Note that unlike the other tools twine should only be run when you actually intend to do a release. This means I don't recommend putting it in tox.ini and instead running it manually. When you become more experienced this process will end up as part of a Continuous Integration / Continuous Deployment (CI/CD) automation pipeline. So now it's time to upload our package:



> pdm build
> twine upload -r testpypi dist/*

First we're building a package to upload which goes into the dist/ directory. Then we tell twine to upload the files in that directory. The -r testpypi tells twine to use the testpypi section we added to .pypirc earlier so it knows where to upload to. This is important as there are actually several potential candidates for uploading python packages to (private repositories such as AWS CodeArtifact and JFrog Artifactory for example). After the upload is complete you will see something like:



View at:
https://test.pypi.org/project/my-pdm-project-cwprogram-test/1.0.0/

Going to this page will show you the project page and how to install the package. I'll make a virtual environment real quick to show the installation:



> python3.11 -m venv pip_test_install_pypi
> pip_test_install_pypi/Scripts/activate
> pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ my-pdm-project-cwprogram-test 
Looking in indexes: https://test.pypi.org/simple/, https://pypi.org/simple/
Collecting my-pdm-project-cwprogram-test
  Obtaining dependency information for my-pdm-project-cwprogram-test from https://test-files.pythonhosted.org/packages/de/82/cce488c69f576b1ba270d5efc91381010f4a0bbbdf17fb6837c81adeb47b/my_pdm_project_cwprogram_test-1.0.0-py3-none-any.whl.metadata
  Using cached https://test-files.pythonhosted.org/packages/de/82/cce488c69f576b1ba270d5efc91381010f4a0bbbdf17fb6837c81adeb47b/my_pdm_project_cwprogram_test-1.0.0-py3-none-any.whl.metadata (274 bytes)
Collecting numpy>=1.25.2 (from my-pdm-project-cwprogram-test)
  Obtaining dependency information for numpy>=1.25.2 from https://files.pythonhosted.org/packages/da/3c/3ff05c2855eee52588f489a4e607e4a61699a0742aa03ccf641c77f9eb0a/numpy-1.26.2-cp311-cp311-win_amd64.whl.metadata
<snip>
Successfully installed certifi-2023.11.17 charset-normalizer-3.3.2 idna-3.6 my-pdm-project-cwprogram-test-1.0.0 numpy-1.26.2 requests-2.31.0 urllib3-2.1.0

So this installation method differs from the one shown on the project page in that it adds the actual PyPI repository as another package source since we're overriding it. Without this the package wouldn't install as it wouldn't be able to find the proper version of numpy.

Adding Metadata

As it is right now the package page is pretty bland because we haven't given enough information to describe our package. So it's time to open up pyproject.toml to add in a few things:



[project]
name = "my-pdm-project-cwprogram-test"
version = "1.0.1"
description = "A tutorial package for building python projects with PDM"
keywords = ["tutorial", "pdm"]
authors = [
    {name = "Chris White", email = "me@nospam.com.com"},
]
dependencies = [
    "numpy>=1.25.2",
    "requests>=2.31.0",
]
requires-python = ">=3.11"
readme = "README.md"
license = {text = "MIT"}

[project.urls]
Homepage = "https://dev.to/cwprogram"

Also the version needs to be updated to 1.0.1 in docs/source/conf.py. So we've updated the version and description as basic information. There's also a new keywords field which is useful for when a user is searching for packages. [project.urls] section was added to give a URL for the project (in this case it's just my dev.to page as there's no real project page). Now there's one more piece of useful information that can be added: classifiers.

Classifiers

PyPI and other packaging formats have the concept of classifiers. There's an exhausting yet comprehensive list on the pypi site. You can use it to tell users about what python your using, environmental constraints, integration with popular packages, operating systems, etc. Let's take a look at some of them we'll use:

Development Status :: 5 - Production/Stable: This is considered a stable release with unit tests, linting, etc.
Operating System :: OS Independent: The package doesn't use graphical interfaces or anything special and as such can run on major operating systems
Intended Audience :: Developers: The intended audience is developers who are looking to create their own python packages
License :: OSI Approved :: MIT License: Technically the license field already covers this but we'll let people know it's an MIT license as well
Programming Language :: Python :: 3.11: This project supports Python 3.11
Topic :: Education :: Testing: This is for educational purposes as a test project

The categories are good enough for our purposes. Once you get an actual project going I encourage you to look into these classifiers along with the keywords to make your project more easily discoverable by end users. Now for our finished product:



[project]
name = "my-pdm-project-cwprogram-test"
version = "1.0.1"
description = "A tutorial package for building python projects with PDM"
keywords = ["tutorial", "pdm"]
authors = [
    {name = "Chris White", email = "me@nospam.com"},
]
dependencies = [
    "numpy>=1.25.2",
    "requests>=2.31.0",
]
requires-python = ">=3.11"
readme = "README.md"
license = {text = "MIT"}
classifiers = [
  "Development Status :: 5 - Production/Stable",
  "Operating System :: OS Independent",
  "Intended Audience :: Developers",
  "License :: OSI Approved :: MIT License",
  "Programming Language :: Python :: 3.11",
  "Topic :: Education :: Testing"
]

[project.urls]
Homepage = "https://dev.to/cwprogram"

Finally re-uploading our new code:



> pdm build
> twine upload -r testpypi dist/*

Now if we look at the newer page you'll notice some differences:

So we have a reasonable description shown, our keywords are present, there's a homepage to point users to, and classifiers are available to give the user more metadata about our project. There are a few other fields and some ways to write fields differently, so I recommend looking over the official documentation on writing pyproject.toml for more information.

Conclusion

This ends the beginners python series, and it's been a fun ride. It took about two months of work and I hope it helps beginners out there get a start on setting up python projects in a fairly modern way. Once your more comfortable I recommend looking into:

GitHub to share your code
pre-commit to automate your tox runs with your git workflow
GitHub Actions for implementing CI/CD to automate your releases
readthedocs for hosting your sphinx documentation

Mastering these I would consider a step into being in an early intermediate level as a software developer. I hope you enjoyed this series and please look forward to more articles in the future! If you like what you see please check my dev.to profile as I'm open for work opportunities.

Automating Your Python Dev Tools With Tox

Chris White — Sun, 03 Dec 2023 22:34:09 +0000

Tooling Centralization
- Tox Configuration
- Tox Organization
- Tox Factors
Conclusion

So far the code for our beginner's project has become fairly stable and there are linting and testing tools to give confidence in basic functionality. It's almost at the level of being ready to release to the public. Before we do this, however, we need to fix one underlying issue to future project development: running all our tools manually. In this article we'll be looking at tox to centralize our tooling runs.

Tooling Centralization

So right now we have the following functionality as part of our python project:

flake8: generalized linting
pylint: more in depth linting
pytest: test suite and test coverage
sphinx: document generation

This is also the order we want to execute them as well. Running each of these manually every time isn't very efficient. To deal with this issue we'll be using the python tool tox. As tox is a development tool we'll need to install it as a traditional dev dependency with pdm. Along with that, there's actually a python package to make working with tox easier in a pdm project. I'll go ahead and install both:

> pdm add -dG dev tox tox-pdm

Tox Configuration

Now tox handles its configuration via an ini file called tox.ini. Let's go ahead and build this up slowly for each of the tools mentioned above. To start out create the tox.ini file in your project's toplevel directory. Then add the following:

[tox]
env_list = py311

This is declaring an environment that will be run under python 3.11. It's used on the backend for setting up a virtual environment. Next we'll setup the tests:

[tox]
env_list = py311

[testenv]
groups = dev
commands =
    flake8
    pylint --recursive=y .
    pytest --cov=my_pdm_project

We'll get to the docs in a later part as that requires an additional change. As far as the layout goes testenv is the default settings that tox uses for all environments. In this case the default is to install dev group dependencies and run the listed commands. env_list is something we'll look at in a bit but lets us diversify what we want to run. The current setting simply indicates that we wish to run the commands using python 3.11, shorthanded to py311.

Now before actually running anything tox will create a .tox folder when executed that acts much like our .venv folder. So this means we'll need to update flake8 and pylint to ignore it. Open up .flake8 in your project root directory and change the content to:

[flake8]
max-line-length = 99
exclude =
  .venv/*
  .tox/*
  docs/*

Then open up pyproject.toml to update this section to add the ignore as well:

[tool.pylint.MASTER]
ignore-paths = [ "^.venv/.*$", "^.tox/.*$", "^docs/*" ]

Note that I've also added docs/ because sphinx from our last article also uses python which we don't have much control over. Now for the moment of truth:

> pdm run tox
py311: commands[0]> flake8
py311: commands[1]> pylint --recursive=y .

--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)

py311: commands[2]> pytest --cov=my_pdm_project
<snip>
py311: OK (6.58=setup[3.38]+cmd[0.55,2.11,0.55] seconds)
congratulations :) (6.66 seconds)

I've focused the output but you can see tox is running all the normal commands for us. All of this is condensed into a single easy to run command!

Tox Organization

Now while everything is centralized, there's a few issues that can come up in larger projects:

Sometimes you just want to run only linting or only tests
Linting doesn't require the package to be installed, which tox will do every time

Fortunately for us there's a way to deal with it. tox supports the concept of multiple environments. That's why the first item is env_list. It also supports installation groups that we've defined in pyproject.toml (in this case the dev group with all of our dev tools). Up until now I've recommended installing everything that's not for the base code to be installed in the dev group for simplicity purposes. Now that we've come further in the project it's time to organize these for making our tox runs cleaner. Right now everything looks like this (save the versions of packages might be different):

[tool.pdm.dev-dependencies]
dev = [
    "flake8>=6.1.0",
    "pylint>=3.0.1",
    "pytest>=7.4.2",
    "pytest-cov>=4.1.0",
    "requests-mock>=1.11.0",
    "sphinx>=7.2.6",
    "tox>=4.11.4",
    "tox-pdm>=0.7.0",
]

tox and tox-pdm are fine as-is in the dev group. The rest we'll break up into linting, testing, and doc. Simply add each group as group_name = [] with the respective version entries inside. Let's do linting first:

[tool.pdm.dev-dependencies]
dev = [
    "pytest>=7.4.2",
    "pytest-cov>=4.1.0",
    "requests-mock>=1.11.0",
    "sphinx>=7.2.6",
    "tox>=4.11.4",
    "tox-pdm>=0.7.0",
]
lint = [
    "flake8>=6.1.0",
    "pylint>=3.0.1",
]

So here pylint and flake8 are now in a dedicate lint group. Now we'll do the same for testing and docs:

[tool.pdm.dev-dependencies]
dev = [
    "tox>=4.11.4",
    "tox-pdm>=0.7.0",
]
lint = [
    "flake8>=6.1.0",
    "pylint>=3.0.1",
]
testing = [
    "pytest>=7.4.2",
    "pytest-cov>=4.1.0",
    "requests-mock>=1.11.0",
]
docs = [
    "sphinx>=7.2.6",
]

As tox references the pdm.lock file, it will need to have all the groups refreshed that is listed within it. This can be done with:

> pdm lock -G:all

Now in tox.ini it's time to break everything up. We'll get everything we have currently updated first:

[tox]
env_list = lint, test

[testenv:lint]
groups = testing, lint
commands =
  flake8
  pylint --recursive=y .

[testenv:test]
groups = testing
commands =
  pytest --cov=my_pdm_project

The major difference here is that we have a testenv:lint and testenv:test. This is known as a "named environment". It's useful for cases of breaking out specific functionality. env_list will run in the order provided with lint first followed by test. For linting the reason why the testing packages is required is because pylint is running against our tests and as such needs to be able to resolve the modules we're using. Let's try this out with a quick run:

> pdm run tox
lint: install_deps> pdm sync --no-self --group testing --group lint
lint: commands[0]> flake8
lint: commands[1]> pylint --recursive=y .

--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)

lint: OK ✔ in 6.22 seconds
test: install_deps> pdm sync --no-self --group testing
test: commands[0]> pytest --cov=my_pdm_project
================================================= test session starts =================================================
tests\test_mymath.py .........                                                                                   [100%]

---------- coverage: platform win32, python 3.11.5-final-0 -----------
Name                                                     Stmts   Miss  Cover
----------------------------------------------------------------------------
.tox\test\Lib\site-packages\my_pdm_project\__init__.py       0      0   100%
.tox\test\Lib\site-packages\my_pdm_project\mymath.py        25      0   100%
----------------------------------------------------------------------------
TOTAL                                                       25      0   100%


================================================== 9 passed in 0.17s ==================================================
  lint: OK (6.22=setup[3.17]+cmd[0.55,2.50] seconds)
  test: OK (3.31=setup[2.75]+cmd[0.56] seconds)
  congratulations :) (9.61 seconds)

Again I've reduced the output to showcase the important parts. Since everything is broken up into separate parts, we can even choose to run only linting for example:

> pdm run tox -e lint
lint: install_deps> pdm sync --no-self --group testing --group lint
lint: commands[0]> flake8
lint: commands[1]> pylint --recursive=y .

--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)

  lint: OK (4.53=setup[0.86]+cmd[0.56,3.11] seconds)
  congratulations :) (4.62 seconds)

This is extremely useful for working on different parts of the development phase where you've passed linting but are trying to just get the tests to pass. Then once the tests are fixed you can run the whole suite to make sure everything as a whole looks fine. For the documentation generation it's slightly similar except we need to enter the docs directory first:

[tox]
env_list = lint, test, docs

[testenv:lint]
groups = testing, lint
commands =
  flake8
  pylint --recursive=y .

[testenv:test]
groups = testing
commands =
  pytest --cov=my_pdm_project

[testenv:docs]
groups = docs
changedir = docs
commands = sphinx-build source/ build/

Sphinx itself is installed thanks to be part of the docs group we setup in pyproject.toml. Running again we can see docs are now being generated:

docs: commands[0] C:\Users\johnsmith\my-pdm-project\docs> sphinx-build source/ build/
Running Sphinx v7.2.6
loading pickled environment... done
building [mo]: targets for 0 po files that are out of date
writing output...
building [html]: targets for 2 source files that are out of date
updating environment: 0 added, 0 changed, 0 removed
reading sources...
looking for now-outdated files... none found
preparing documents... done
copying assets... copying static files... done
copying extra files... done
done
writing output... [100%] mymath
generating indices... genindex py-modindex done
writing additional pages... search done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded.

The HTML pages are in build

Another issue here is that while sphinx and pytest need our package installed to work properly, linting is just working against the code itself. This means installation of our package just adds unnecessary time. We can skip the process by using skip_install = true in our lint section:

[testenv:lint]
groups = testing, lint
skip_install = true
commands =
  flake8
  pylint --recursive=y .

Running tox again shows it's no longer installing our package for the linting session:

> pdm run tox
lint: install_deps> pdm sync --no-self --group testing --group lint
lint: commands[0]> flake8
lint: commands[1]> pylint --recursive=y .

--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)

lint: OK ✔ in 4.59 seconds

Now unlike the previous definition where we had py311 there's nothing here indicating which version of python is used. In this case tox simply uses the one from the current environment (python 3.11 in this case, which is what we created the pdm virtual environment with). It turns out you can be explicit with python versions by utilizing an interesting feature called factors.

Tox Factors

This is somewhat of a fancy way of saying "names separated by hyphens", with a twist when it comes to python versions. I'm going to go ahead and make an isolated test case to showcase this:

[tox]
env_list = py311, py3.12

[testenv]
commands = python --version

Now to go ahead and run this:

> pdm run tox
py311: commands[0]> python --version
Python 3.11.5
py3.12: commands[0]> python --version
Python 3.12.0

So what's happening is that the format pyMajorMinor and pyMajor.Minor are special cases called "default factors" and they map to specific python versions. In this case py311 maps to python 3.11 and py3.12 maps to python 3.12. The tox documentation has the full details on special python interpreter factors. I will note that this only works if both python 3.11 and python 3.12 are installed on your system. Multiple python versions is an interesting topic, though might be somewhat more of an intermediate discussion. I've written about here if you're interested. This isn't just for python versions though, you can even get a bit more advanced by testing against multiple environment combinations. Let's say for example you made a webapp that you want to ensure works against database A and database B:

[tox]
env_list = py311-databaseA, py3.12-databaseB

[testenv]
commands = 
  python --version
  databaseA: python -c 'print("uses databaseA")'
  databaseB: python -c 'print("uses databaseB")'

So let's see what happens:

> pdm run tox
py311-databaseA: commands[0]> python --version
Python 3.11.5
py311-databaseA: commands[1]> python -c "print(\"uses databaseA\")"
uses databaseA
py311-databaseA: OK ✔ in 3.47 seconds

py3.12-databaseB: commands[0]> python --version
Python 3.12.0
py3.12-databaseB: commands[1]> python -c "print(\"uses databaseB\")"
uses databaseB

This is because what's actually happening is:

py311-databaseA = factor py311 + factor databaseA
py3.12-databaseB = factor py3.12 + factor databaseB

[testenv]
commands = 
  python --version
  databaseA: python -c 'print("uses databaseA")'
  databaseB: python -c 'print("uses databaseB")'

So here we can target specific factors, even if all are not included, to take on specific commands or install specific package dependencies. This makes tox an incredibly powerful tool. Even so our current needs are simple so leaving out the python version factors is fine as we're only testing the version set by pdm. Once you get farther in your python development career you'll be able to better understand how powerful the more advanced usages are.

Conclusion

My original intention was to combine this with uploading your package via twine. Then when I started writing it came to realization that the amount of explanation required would have made it a bit too verbose for my liking. I'd also like to note that I'm currently open for work if you like what you see. To not sound too spammy I'll just say check my dev.to profile for more information. In the next section we'll see the final installment of this series where we upload our python code for others to use.

Python Versions and Release Cycles

Chris White — Wed, 29 Nov 2023 15:49:11 +0000

Python 2 and 3
Python Version Components
Python Release Cycle
- End Of Life
- Security
- Bugfix
- Feature
Which Version To Use
- Beginners
- Companies
- Package Maintainers
Supporting Multiple Versions
Migrating Versions
Conclusion

As with all programming languages, the python has multiple versions and a release cycle to go with it. Understanding this release cycle is crucial as a python developer to reducing tech debt in your stack. This article will look at multiple python versions, release cycles, and making decisions on which version to utilize.

Python 2 and 3

If you've been in python long enough, you might have heard about the fun that is the python 2 to 3 transition. It was a fairly major revamp of the language which required migrations for a substantial portion of the userbase (myself included). It was so much that they ended up making a tool for it. This unfortunately left a sour taste for many developers as these kind of migrations can put pressure on developers employed in companies who are already attempting to meet deadlines. Needless to say it that many developers were not thrilled about it. After many extensions of timelines, python 2 received its end of life on January 1st 2020 with the python 2.7 series. The reason why I'm mentioning all of this is that it's definitely something python developers consider a not so great outcome and has had an impact on how they consider new version / large feature transitions.

Python Version Components

Python versions include a major, minor, and micro component. It's slightly similar in concept to semver, though the minor version may do something incompatible at times. An example of this is the removal of distutils in 3.12. While technically not a language change persay, it did mean some build systems had to be revamped to accommodate this change. In general you'll see python referred to in a major.minor format such as:

Python 2.7
Python 3.9
Python 3.12

New releases and security fixes are where you'll tend to find them referred to in full version format such as:

Python 2.7.17
Python 3.9.18
Python 3.12.0

Security releases use the full format so you know exactly which version(s) are affected. New releases show it so you can tell how many upgrade paths it would take to reach it. Since the release of python 3 in December of 2008 there hasn't been a new major release. This is due to requiring a substantial amount of incompatible changes. There hasn't been an plans announced for a python 4 at this time according to an interview with the language's creator. Though if there was it might be around reworking how the C API is handled which is a pain point for many.

Python Release Cycle

The python dev guide has a good breakdown of the release cycle process. Here is a screenshot of the current release cycle:

As shown here there are quite a number of end-of-life versions, followed by security, then bugfixes, and finally a feature version. All categories are referenced by their respective major.minor version. Let's take a look at the categories and what they entail.

End Of Life

Not much to say here, it basically means the version is not supported by python developers. No security, bugfixes, or feature additions will be added. Both source and binary version updates will no longer be available. You really want to avoid being on an end of life version, especially in corporate environment with security compliance programs.

Security

Security releases are where developers will only add security updates, so there will be no feature updates or non-security related bugfixes. They also won't have binaries available on the official website. You generally want to avoid being on the oldest security release as that means an end-of-life will be in the future.

Bugfix

These versions include fixes for bugs and security related issues. They also have official binaries available for them on the python website. If you're a new developer going for one of the bugfix versions would be ideal since binaries being available make them easier to install. The latest bugfix version would be ideal unless there's a chance that would require updates to popular packages. Bugfixes at some point will become security releases and lose their binary availability after the last bugfix release.

Feature

This is more of a bleeding edge release. It indicates new features planned for the future. As 3.12 was recently released as bugfix, 3.13 is now the feature release. One planned introduction in particular is an optional build mode that removes the global interpreter lock. It's generally recommended for beginners to avoid these since it's more of an in progress release where certain tutorials may be outdated.

Which Version To Use

So now the question becomes, which version should be utilized? Much of the answer lies in how it's utilized. One thing I will say is that no one should be using an end-of-life version (okay, maybe that one script from 2010 you wrote to parse movie titles). Let's take a look at some of the user categories and what version they might use.

Beginners

In general I recommend beginners be on the latest bugfix version. The exception to this is I wouldn't pick up a recent bugfix release if it hasn't been out for at least 4 months (3.12 at time of writing would fall under this). This is because tutorials might need updating, and some popular python packages might need time to migrate to it. It's better to play it safe. The main reason for recommending it is that binaries are widely available which make installing them very easy to pull off. This goes for both Windows and Mac in particular.

Companies

Recent startups may be closer to the bugfix versions as they don't have systems to migrate and being on newer versions is more advantageous tech debt wise. Enterprises generally stay within the security releases. Their focus is ensuring all their systems work as intended without unreasonable changes introduced. Sometimes there may be other limiting factors such as versions available through external solutions providers. AWS for example has a range of python versions for using with AWS lambda. They would want to be on at least 3.8 minimum given that there's a deprecation warning that went out recently for 3.7. There also might be requirements on what third party pip packages are available.

Package Maintainers

In general package maintainers will decide which versions to support based on what their userbase looks like and how many people are motivated/available to maintain said versions. At minimum package maintainers will generally use the oldest security release all the way up to the latest bugfix release (save python 3.12 which required some updates). Packages may require other packages which have their own version constraints and can also restrict which version is supported.

Supporting Multiple Versions

Package maintainers and other individuals may want to support multiple versions of python. For systems such as Windows and MacOSX you can download binaries of the bugfix versions. Security versions on the other hand don't have binaries officially available. In this particular case Linux is nice for working with multiple versions as packages are either installed from source or end binaries are build from source (especially important for security release where only the source is available). Distribution wise there are a few options:

Ubuntu has some bugfixes available along with a deadsnakes PPA. Being non-official the security implications should be considered, but it allows for easily installation of various python versions in a distro friendly way.
Fedora contains a wide range of python packages available through their built-in package manager. You could setup a quick dev box with Fedora on Raspberry Pi as one solution.

For OSX there is homebrew or pyenv (pyenv is another solution on Linux). As pyenv compiles from source it will require setting up XCode (the Apple IDE) tools to support this which can be pretty bulky. Windows users have chocolatey but the issue there is it works off the binaries. That means it won't have the latest security release available since those are source only. Conda is also another solution which can be picked up by Visual Studio Code as available versions of Python making development easier. In the end it might be best to consider using WSL on Windows for installing a Linux version and using that instead.

Migrating Versions

For companies I recommend looking into having upgrade path at regular intervals. The more you upgrade the less of a chance you'll be stuck with an EOL version. It also makes future upgrades easier, or at least gives you a buffer to handle issues. Having a CI/CD pipeline via something such as GitHun Actions along with a test suite makes this process even easier. Simply upgrade your pipeline to the new version then see what the test suite results are. If everything looks good you may be able to upgrade fairly easily. In the case of there being issues, you can at least have the gradual low priority tasks to investigate the issues and handle them at a better time. This is especially important in businesses with high priority deadlines.

Conclusion

This concludes a look into how to work with Python versions. It's something I was planning to go over in my beginners series until I realized it would be better flushed out on its own article. I will note that future articles will have a different direction to them as I'm currently in job hunt mode. This means that the direction will be more towards selling myself as a technically competent individual instead of the usual "I want to mess around with this interesting tech". This includes potentially doing AWS articles that I've tried to avoid here due to not being open source (the stack at least). If you're interested in hiring me for a remote full time position please look at my dev.to profile for more information.

Python Documentation With Docstrings and Sphinx

Chris White — Fri, 10 Nov 2023 17:30:52 +0000

pylint Prep
Docstrings
Documentation Generation With sphinx
Conclusion

In the last installment of the series we looked at how to achieve testing in our python projects. Much like testing, documentation can go a long way towards achieving adoption for your project. In this case we're going to be looking at how to generate documentation of your code using docstrings and sphinx.

pylint Prep

When doing the linting I explicitly disabled checking for docstrings. Now that we'll be using them, it's time to enable that check in the pyproject.toml file. First however, we'll want to check what the current state of our code is:

> pdm run pylint --recursive=y .
************* Module src.my_pdm_project.mymath
src\my_pdm_project\mymath.py:10:4: R1720: Unnecessary "elif" after "raise", remove the leading "el" from "elif" (no-else-raise)
src\my_pdm_project\mymath.py:16:10: W3101: Missing timeout argument for method 'requests.get' can cause your program to hang indefinitely (missing-timeout)
src\my_pdm_project\mymath.py:20:4: R1705: Unnecessary "else" after "return", remove the "else" and de-indent the code inside it (no-else-return)
************* Module tests.test_mymath
tests\test_mymath.py:23:11: C0123: Use isinstance() rather than type() for a typecheck. (unidiomatic-typecheck)
tests\test_mymath.py:24:11: C0123: Use isinstance() rather than type() for a typecheck. (unidiomatic-typecheck)

------------------------------------------------------------------
Your code has been rated at 9.25/10 (previous run: 9.25/10, +0.00)

So we see how the general development process works of writing code, checking with linting, and fixing things that come up. So first off is this section:

    if operation == '/' and b == 0:
        raise ZeroDivisionError
    elif operation not in SUPPORTED_OPERATIONS:
        raise ValueError

This is pretty simple, it just wants an if used instead of elif due to how exceptions handle breaks of logic flow. I'll go ahead and update it here:

    if operation == '/' and b == 0:
        raise ZeroDivisionError
    if operation not in SUPPORTED_OPERATIONS:
        raise ValueError

The next is that requests is being used without setting a timeout value. If the server was not responsive then our connection might end up in a stuck state. I'll go ahead and fix that here:

    res = requests.get(
        f'{BASE_URI}{operation_expression}', timeout=20
    )

This will throw an error if a response is not received within 20 seconds. Next is that the else here is redundant:

    if operation == '/':
        return float(res.text)
    else:
        return int(res.text)

I'll go ahead and update that here:

    if operation == '/':
        return float(res.text)
    return int(res.text)

Finally in our tests we're using type() instead of isinstance() which is cleaner. I'll go ahead and update the tests to do that:

    assert result == 5
    assert isinstance(result, int)
    assert isinstance(result2, float)

Now that everything is cleaned up I'll go ahead and run pylint again:

> pdm run pylint --recursive=y .

-------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 9.25/10, +0.75)

Now it's time to enable the docstring check. I'll go ahead and do this by removing the following portion from pyproject.toml:

[tool.pylint."MESSAGES CONTROL"]
disable = '''
missing-module-docstring,
missing-class-docstring,
missing-function-docstring
'''

Now I'll run pylint again:

> pdm run pylint --recursive=y .
************* Module src.my_pdm_project.mymath
src\my_pdm_project\mymath.py:1:0: C0114: Missing module docstring (missing-module-docstring)
src\my_pdm_project\mymath.py:9:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:25:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:29:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:33:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:37:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:41:0: C0116: Missing function or method docstring (missing-function-docstring)
************* Module tests.test_mymath
tests\test_mymath.py:1:0: C0114: Missing module docstring (missing-module-docstring)
tests\test_mymath.py:16:0: C0116: Missing function or method docstring (missing-function-docstring)
tests\test_mymath.py:27:0: C0116: Missing function or method docstring (missing-function-docstring)
tests\test_mymath.py:32:0: C0116: Missing function or method docstring (missing-function-docstring)
tests\test_mymath.py:39:0: C0116: Missing function or method docstring (missing-function-docstring)
tests\test_mymath.py:46:0: C0116: Missing function or method docstring (missing-function-docstring)
tests\test_mymath.py:52:0: C0116: Missing function or method docstring (missing-function-docstring)
tests\test_mymath.py:62:0: C0116: Missing function or method docstring (missing-function-docstring)

-------------------------------------------------------------------
Your code has been rated at 7.61/10 (previous run: 10.00/10, -2.39)

So as you can see there's a lot of output. Now let's talk about how to go about fixing this.

Docstrings

Docstrings are done by enclosing text in triple double quotes. As an example:

def add_numbers(a: int, b: int):
    """
    This is my function
    """
    return make_mathjs_request(a, b, '+')

Now the official docstring specification is part of PEP 257. While it does describe the overall format it's not specific about the format of what you would put in a doc string. In this case I'm going to be utilizing sphinx as the code documentation generator of choice. Now let's look at what our function's doc string will become using the sphinx format:

def add_numbers(a: int, b: int):
    """Add two numbers together

    :param a: The base integer to use in the add operation
    :type a: int
    :param b: The integer to add to the base integer
    :type b: int

    :return: The sum of both integers
    :rtype: int
    """

:param: indicates what a parameter is meant for and :type: indicates the type of the parameter. :return: will describe the return of the function and :rtype: the return value. Now after implementing this docstring and running pylint again:

> pdm run pylint --recursive=y .
************* Module src.my_pdm_project.mymath
src\my_pdm_project\mymath.py:1:0: C0114: Missing module docstring (missing-module-docstring)
src\my_pdm_project\mymath.py:9:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:39:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:43:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:47:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:51:0: C0116: Missing function or method docstring (missing-function-docstring)
************* Module tests.test_mymath
tests\test_mymath.py:1:0: C0114: Missing module docstring (missing-module-docstring)
tests\test_mymath.py:16:0: C0116: Missing function or method docstring (missing-function-docstring)
tests\test_mymath.py:27:0: C0116: Missing function or method docstring (missing-function-docstring)
tests\test_mymath.py:32:0: C0116: Missing function or method docstring (missing-function-docstring)
tests\test_mymath.py:39:0: C0116: Missing function or method docstring (missing-function-docstring)
tests\test_mymath.py:46:0: C0116: Missing function or method docstring (missing-function-docstring)
tests\test_mymath.py:52:0: C0116: Missing function or method docstring (missing-function-docstring)
tests\test_mymath.py:62:0: C0116: Missing function or method docstring (missing-function-docstring)

------------------------------------------------------------------
Your code has been rated at 7.76/10 (previous run: 7.61/10, +0.15)

We can see there is a slight increase in our overall score since we added the docstring. Now one issue here is that pylint is expecting the tests to have docstrings. Functionality wise we really don't need them in our tests since the point of code documentation is to show the users how the code they're consuming works. A general user isn't going to be consuming tests. At the top of the test file I can tell pylint to ignore docstrings for them since I still want it to make sure the other parts of my tests are solid:

# pylint: disable=missing-docstring
from urllib.parse import quote_plus
import pytest
import requests_mock

from my_pdm_project.mymath import (
    add_numbers,

Now after running pylint:

> pdm run pylint --recursive=y .
************* Module src.my_pdm_project.mymath
src\my_pdm_project\mymath.py:1:0: C0114: Missing module docstring (missing-module-docstring)
src\my_pdm_project\mymath.py:9:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:39:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:43:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:47:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:51:0: C0116: Missing function or method docstring (missing-function-docstring)

------------------------------------------------------------------
Your code has been rated at 9.05/10 (previous run: 8.96/10, +0.09)

So I'm a big closer here. Now besides the functions it's also mentioning a module docstring. At the top of our python code we can simply write a description of what the underlying code is meant to do:

"""
A module containing simple math operations.
"""
from urllib.parse import quote_plus
import numpy as np
import requests

Now another check:

> pdm run pylint --recursive=y .
************* Module my_pdm_project.mymath
src\my_pdm_project\mymath.py:12:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:42:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:46:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:50:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:54:0: C0116: Missing function or method docstring (missing-function-docstring)

------------------------------------------------------------------
Your code has been rated at 9.21/10 (previous run: 9.05/10, +0.16)

So now just the function docstrings are left. Let's look at a more complex example for make_mathjs_request:

def make_mathjs_request(a: int, b: int, operation: str):
    """Make a expression call against the MathJS API

    :param a: Base integer for the operation
    :type a: int
    :param b: Integer to use with a in the operation
    :type b: int
    :param operation: Operation to run against a and b
    :type operation: str

    :raises ZeroDivisionError: Raised if division by 0
    :raises ValueError: Raised if not a supported operation

    :returns:
        - int for non division operations
        - float for division operations
    """

Here it's somewhat like what we saw before. What's new now is that we also documentation exceptions that can be raised, and why they'd be raised. The :returns: is used since we're returning either a float or int depending on the operation. As mentioned before this was done as an example case for showing testing and normally you wouldn't want a function to return multiple types. After a pylint run:

> pdm run pylint --recursive=y .
************* Module my_pdm_project.mymath
src\my_pdm_project\mymath.py:58:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:62:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:66:0: C0116: Missing function or method docstring (missing-function-docstring)
src\my_pdm_project\mymath.py:70:0: C0116: Missing function or method docstring (missing-function-docstring)

------------------------------------------------------------------
Your code has been rated at 9.37/10 (previous run: 9.21/10, +0.16)

So I just need to add docstrings to the other functions. After all is done the file looks like this:

"""
A module containing simple math operations.
"""
from urllib.parse import quote_plus
import numpy as np
import requests

BASE_URI = "http://api.mathjs.org/v4/?expr="
SUPPORTED_OPERATIONS = ['+', '-', '*', '/']


def make_mathjs_request(a: int, b: int, operation: str):
    """Make a expression call against the MathJS API

    :param a: Base integer for the operation
    :type a: int
    :param b: Integer to use with a in the operation
    :type b: int
    :param operation: Operation to run against a and b
    :type operation: str

    :raises ZeroDivisionError: Raised if division by 0
    :raises ValueError: Raised if not a supported operation

    :returns:
        - int for non division operations
        - float for division operations
    """
    if operation == '/' and b == 0:
        raise ZeroDivisionError
    if operation not in SUPPORTED_OPERATIONS:
        raise ValueError

    operation_expression = quote_plus(f'{a}{operation}{b}')
    res = requests.get(
        f'{BASE_URI}{operation_expression}', timeout=20
    )

    if operation == '/':
        return float(res.text)
    return int(res.text)


def add_numbers(a: int, b: int):
    """Add two numbers together

    :param a: The base integer to use in the add operation
    :type a: int
    :param b: The integer to add to the base integer
    :type b: int

    :return: The sum of both integers
    :rtype: int
    """
    return make_mathjs_request(a, b, '+')


def subtract_numbers(a: int, b: int):
    """Subtract two numbers

    :param a: The base integer to use in the subtract operation
    :type a: int
    :param b: The integer to subtract the base integer from
    :type b: int

    :return: The subtraction of both numbers
    :rtype: int
    """
    return make_mathjs_request(a, b, '-')


def multiply_numbers(a: int, b: int):
    """Multiple two numbers together

    :param a: The base integer to use in the multiply operation
    :type a: int
    :param b: The integer to multiply against the base number
    :type b: int

    :return: The result of multiplying both numbers
    :rtype: int
    """
    return make_mathjs_request(a, b, '*')


def divide_numbers(a: int, b: int):
    """Divide two numbers

    :param a: The base integer to use in the add operation
    :type a: int
    :param b: The integer divide a by
    :type b: int

    :return: The quotient of the division operation
    :rtype: float
    """
    return make_mathjs_request(a, b, '/')


def average_numbers(numbers: list[int]):
    """Average a list of numbers

    :param numbers: The list of numbers to average
    :type numbers: list[int]

    :return: The average of the numbers
    :rtype: float
    """
    return np.average(numbers)

After finishing up documenting everything here we can check what pylint has to say:

> pdm run pylint --recursive=y .

-------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 9.37/10, +0.63)

Everything is in the clear now!

Documentation Generation With sphinx

Now as is the code documentation is a bit difficult to work with for an average user. To help with this we can use sphinx to take our docstrings and generate them into various formats. As with our other tools this will be added as a development package:

> pdm add -dG dev sphinx

Now we'll need to create a document directory for where our documentation will be stored.

> mkdir docs

Now it's time to setup sphinx to generate documentation. The thing to keep in mind with sphinx is it's primarily a documentation generation tool and generation of library documentation is a side bonus. With this in mind we'll go ahead and setup how our project will work vi the nice sphinx-quickstart utility:

> cd docs
> pdm run sphinx-quickstart --no-makefile -M --ext-autodoc -p "my-pdm-project" -a "Chris White" -v "0.3.0" -r "0.3.0" -l "en" --sep .

So there's a few things to digest here. --no-makefile and -M are done to avoid using make for building. This was mostly to avoid adding in another thing to install. -p sets the name of the project, -a the author, -v and -r are for version and release. They're the same right now because there's no 1.0 release yet, but if there was I'd recommend something like 1.0 for the version and 1.0.0 for the release. It's somewhat like how there's 3.12 for python but the 3.12 version has several releases under it. -l sets the language of the project and --sep ensures that source and build directories are separate. I tend to prefer this because it's easier to ignore the build directory through things like .gitignore later on. Finally . indicates the directory of the project, or more specifically the "documentation project" (as supposed to the code project where all our code is). Now that everything is setup we can use sphinx-build to generate html for us:

> pdm run sphinx-build source/ build/

Now if I look in the build directory there will be an index.html I can access via a browser:

Right now there isn't much going on and "Module Index" doesn't work because it hasn't been setup to recognize our docstrings. To do this we'll create a new file in docs/source/ called mymath.rst with the following content:

Mymath Module Documentation
===========================
.. automodule:: my_pdm_project.mymath
    :members:

Now this format is known as rst or reStructuredText. It's a format that's more feature rich than markdown and useful for structured documentation. In this case it's referring to a function in rst. These functions are in the form:

.. function_name:: arguments
    :option: value

    content

In the case of automodule it will generate documentation for my_pdm_project.mymath including all of its members. Now in the same directory there will be an index.rst file that needs to be edited like so:

.. my-pdm-project documentation master file, created by
   sphinx-quickstart on Fri Nov 10 09:01:15 2023.
   You can adapt this file completely to your liking, but it should at least
   contain the root `toctree` directive.

Welcome to my-pdm-project's documentation!
==========================================
.. toctree::
   :maxdepth: 2
   :caption: Contents:

   mymath


Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

What's changed is that mymath has now been added to our table of contents. Note that contents are indicated by a blank line after the options. In this case the content is mymath on a single line. sphinx knows that this is referencing mymath.rst and toctree will automatically parse mymath.rst to provide a table of contents. Now after running this again in the docs directory:

> pdm run sphinx-build source/ build/

The main index.html page will show:

And clicking on "Mymath Module Documentation" will show the documentation generated via our doc strings:

Conclusion

This concludes a look at documentation generation via sphinx parsing python docstrings. I will say that rst is more involved than simple markdown, but it's feature rich nature makes it ideal for many forms of documentation structure. In the next section we'll be looking at orchestrating all of our tools so far and uploading our code for everyone to use.