Forem: Aaron Maxwell

Levels of Python type annotations

Aaron Maxwell — Mon, 04 May 2026 16:30:00 +0000

Python has optional type annotations - also called "type hints". Like this:

def entry_to_dict(entry: Entry) -> dict:
    return {
        'title': entry.title,
        'num_likes': entry.num_likes,
        'url': entry.url,
        }

The annotations here being "Entry" as the type for the "entry" argument, and "dict" as the return type.

In fact, there are at least 3 ways type annotations can be used:

documentation
to configure libraries and tools
static type checking

These are so different, asking "do you use type annotations?" is really too vague. It's three separate questions.

The first is demonstrated by entry_to_dict() above. You are reading the code, and just by reading it, you know that entry should be an instance of a class called Entry, and entry_to_dict() should return a dictionary.

This by itself is really useful. And part of what makes it useful is how easy it is to include when you write the code. There's very little "friction" to dropping these types in as you bang out the method definition... And you can just skip them when they are too complex (or unknown) to specify.

(Bonus: if you're the type of developer who uses autocomplete in IDEs, type hints can improve its capabilities in some cases.)

The second case - configuring libraries and tools - is demonstrated in data classes:

@dataclass
class Entry:
    email: str
    when: str

    @classmethod
    def from_csv_row(cls, row):
        email = row['Customer email'].lower()
        when = parse_scheduleonce_date(row["Meeting date and time in Owner's time zone"])
        return cls(email, when)

    def __hash__(self):
        return hash( (self.email, self.when) )

See the "email: str" and "when: str"? @dataclass uses those annotations to do its magic.

The third use case, static type checking, is a different animal.

Because statically typed languages (like Java) have an advantage over Python: the type of every variable and expression is predefined, so the compiler can catch many errors at compile time that Python cannot...

In fact, Python itself cannot catch those until the line of code is actually executed...

And the promise of "static type checking" is to fix that. Using a tool called mypy.

To get the full benefits, though, requires an ongoing tax on your time.

You must essentially learn a new mini-language to specify types, with constructs like Optional[List[str]] and Tuple[_T, _T] that just look weird until you learn how they work. Then invest effort with every function, class, and method you write, to specify the annotations correctly, consistently, and thoroughly.

Which takes more effort than you would like, frankly. It becomes an ongoing tax on your codebase, paid in your attention while coding.

But the larger the Python application, the greater the payoff. And especially for larger, enterprise-grade software systems, the benefits start to greatly outweigh the costs.

If you liked this, you will enjoy the Powerful Python Newsletter.

Pandas' secret mini-language

Aaron Maxwell — Sun, 03 May 2026 12:45:00 +0000

The DataFrame class (from Pandas) is a work of art. Even if you never "do data", priceless lessons can be gleaned by studying this class.

It starts simple enough. Usually you will create a DataFrame by ingesting from a CSV file or database table or something. But you can whip up a small one like this:

import pandas as pd

df = pd.DataFrame({
        'A': [-137, 22, -3, 4, 5],
        'B': [10, 11, 121, 13, 14],
        'C': [3, 6, 91, 12, 15],
    })

This gives you a DataFrame with three columns, labeled A, B and C. With rows of data, like so:

>>> print(df)
     A    B   C
0 -137   10   3
1   22   11   6
2   -3  121  91
3    4   13  12
4    5   14  15

The first thing to notice is that DataFrame is a class. Once upon a time, there was no such thing as a DataFrame. Someone imagined it, and then coded it up. And just look how it changed the world.

If that is not an argument for learning OOP, I do not know what is.

But this article is about something else. Because once you have a DataFrame, you can 'filter' out rows that match certain criteria, by typing magick chars inside square brackets:

>>> positive_a = df[df.A > 0]
>>> print(positive_a)
    A   B   C
1  22  11   6
3   4  13  12
4   5  14  15

Breaking this down:

You can refer to column A of the DataFrame "df" by this handle: df.A
You can use square brackets, e.g. "df[...]", to say: "give me a new DataFrame with only certain rows, skipping others."
What rows are kept? Well, that gets defined by an expression like "df.A > 0" in the brackets. Meaning, in this case: keep those rows where the 'A' value is positive.

And it is interesting that you can go much more complex:

# Relationships between multiple columns:
df[df.A + df.B == 19]
df[d.B - d.C >= 3]

# Logical "and" and "or":
df[(df.A > 0) & (df.B >= 12)]
df[(df.A >= 3) | (df.B == 11)]

# Equation-like filters, e.g.:
df[df.C + 2 < df.B]

Which is super cool.

But it also makes no sense, when you think about it...

Because "df.A > 0" evals as boolean True or False. Right? Exactly one bit of information. So how in the name of Guido does this work?

It works because "df.A > 0" is not boolean. It sneakily does not return True, nor False. It evaluates to something else:

>>> comparison = (df.A > 0)
>>> type(comparison)
<class 'pandas.core.series.Series'>
>>> print(comparison)
0    False
1     True
2    False
3     True
4     True
Name: A, dtype: bool

Well how about that. Rather than bool, its type is something called Series. Which is kinda like a list of bool's, one for each row. Now it makes more sense, doesn't it.

This is the core principle which allows DataFrame's square bracket trick to work.

And while I do not have room in this little essay to fully explain the details...

I will point out that it relies on Python's particular object model. Meaning, it relies on OOP. But also on details of how Python uniquely does OOP. And when you invest in mastering OOP in Python, it helps you make powerful classes like DataFrame too.

Something to think about.

Of course, this is an advanced technique. But see what good ideas all this sparks for you. How can you apply what you learned today in your own code?

(To get a hint to how (df.A > 0) "returns" something other than a bool, search Python's docs for this magic method name: __gt__().)

If you liked this, you will enjoy the Powerful Python Newsletter.

Sneaky version control trick

Aaron Maxwell — Sat, 02 May 2026 12:45:00 +0000

I was writing a program to process some CSV data, exported from a vendor. And it has a date-time column in some creative, funky format.

To help with ingesting this into a Pandas dataframe, I wrote a converter function: it takes that string, and returns a Python datetime object. Simple, right?

But you know, this is a PERFECT place to write a unit test. And I decided to use Pytest, starting with this simple test:

from datetime import datetime
from myprog import parse_vendor_datetime

def test_parse_vendor_datetime():
    actual = parse_vendor_datetime('12/25/38 1:14am EST')
    expected = datetime(2038, 12, 25, 1, 14)
    assert expected == actual

I run the test, and it fails, like it should. Then I write parse_vendor_datetime(), and the test passes. Great!

Now, what no one ever talks about is how TDD works with version control. The secret is that there's actually a really productive way to do it.

Because after I check this test-passing code into version control, in my private branch... the next thing I do is run my program on some real data. And I find it doesn't handle time zones quite right.

Okay... so here's what I do. I get an example of the problematic data, and add to my test function:

def test_parse_vendor_datetime():
    actual = parse_vendor_datetime('12/25/38 1:14am EST')
    expected = datetime(2038, 12, 25, 1, 14)
    assert expected == actual

    actual = parse_vendor_datetime('08/17/38 11:05pm EDT')
    expected = datetime(2038, 8, 17, 11, 5)
    assert expected == actual

I run it, and verify it fails like it should. And I check THAT into version control - again, in my private branch. Because when I have a correctly failing test for a bug, that's progress. Even if that test isn't passing yet.

But it bugs me that this code is repetitive. Doesn't it bug you? Ugh. Makes my skin crawl, just looking at it.

So I refactor it, to use something Pytest calls "parametrized tests":

@pytest.mark.parametrize(
    'dt_str,expected', [
    ('12/25/38 1:14am EST', datetime(2038, 12, 25, 1, 14)),
    ('08/17/38 11:05pm EDT', datetime(2038, 8, 17, 11, 5)),
    ])
def test_parse_vendor_datetime(dt_str, expected):
    actual = parse_vendor_datetime(dt_str)
    assert expected == actual

See how the test code is not repetitive anymore? Even if you don't now what this "parametrize()" thing is, because I haven't explained it, you still understand this is an improvement. If I need to change something, or add another test case, I only need to change the code in one place.

So now, I check THIS into version control. The test is still failing. But changing the code to be less repetitive - more maintainable - is still forward progress, and worthy of checkpointing.

And: Because I had checked in the code before it too, if I mess up the test somehow, I can always roll back to the point it was correct.

What's fascinating:

When you build on this process, and add a few more ingredients... there's a COGNITIVE benefit.

It's potent. You can actually get into a state of flow very easily. In a way that's easy to maintain, and to keep your focus, and become extremely productive.

Having a blast, as you continue cranking out code. Riding that wave for as long as you want. It comes out of how you use version control along with writing automated tests, and doing test-driven development.

If you liked this, you will enjoy the Powerful Python Newsletter.