Forem: Tushar Sadhwani

The math behind Python's slices

Tushar Sadhwani — Thu, 02 Sep 2021 17:23:29 +0000

Read the original post on my blog

The basics

Let's get the basics out of the way first:

Slicing in Python is a way to easily extract a section of a list.

The simplest form of a slice only considers the first two parts, a start index and an end index. Not providing either the start or the end will result in you getting all the numbers from the beginning and/or the end.

Like so:

>>> nums = [1, 2, 3, 4, 5, 6]
>>> nums[1:3]
[2, 3]
>>> nums[:3]
[1, 2, 3]
>>> nums[3:]
[4, 5, 6]

Note that the end index, when provided, is never included in the result.

And just for completion's sake, if you don't provide either, it just clones the entire list:

>>> nums[:]
[1, 2, 3, 4, 5, 6]

Apart from these, there's also a third argument: step, which tells how many numbers it should increment its index by, to get to the next element. step is 1 by default.

>>> nums = [1, 2, 3, 4, 5, 6]
>>> nums[::1]
[1, 2, 3, 4, 5, 6]
>>> nums[::2]
[1, 3, 5]
>>> nums[::4]
[1, 5]

The interesting bits

You can imagine the slicing algorithm being used by the interpreter to be as following:

def slice(array, start, stop, step=1):
    result = []
    index = start
    while index < stop:
        result.append(array[index])
        index += step

    return result

This explains the behaviour of end never being included, and how step decides how to pick the next value.

But, how about this:

>>> nums[:-1]
[1, 2, 3, 4, 5]
>>> nums[:-3]
[1, 2, 3]
>>> nums[-3:-1]
[4, 5]
>>> nums[-1:-3:-1]
[6, 5]
>>> nums[-1:-3]
[]

What's going on in here?

Negative numbers in slices

It should be common knowledge that you can provide negative indices in Python to get a number from the end:

>>> nums = [1, 2, 3, 4, 5, 6]
>>> nums[-1]  # last index
6
>>> nums[-2]  # second from the end
5
>>> nums[len(nums)-2]  # it's the same thing
5

Well, the same thing happens in slices as well:

If you give it a negative start or stop value, it will be treated as that same index from the end.

Like, all of these 3 mean the same thing:

>>> nums[  3 :   5]
[4, 5]
>>> nums[6-3 : 6-1]
[4, 5]
>>> nums[ -3 :  -1]
[4, 5]

And once you know this, it's simple math.

For example:

>>> nums[:-1]    # all values except the last one
[1, 2, 3, 4, 5]
>>> nums[:-3]    # all values except the last three
[1, 2, 3]
>>> nums[-3:]  # all values from last 3rd
[4, 5]

And here's an updated slice Python function that factors this in:

def slice(array, start, stop, step=1):
    if start < 0:
        start = len(array) + start
    if stop < 0:
        stop = len(array) + stop

    result = []
    index = start
    while index < stop:
        result.append(array[index])
        index += step

    return result

Negative `step`

Now the only thing we haven't covered in the examples above is a negative step value. I'm sure you must have seen this one rather un-intuitive way to reverse a list in Python:

>>> nums[::-1]
[6, 5, 4, 3, 2, 1]

What's going on here?

Well, essentially whenever the step value is negative, Python starts iterating from behind. Essentially, the default start value becomes the end of the array and the default stop value becomes the start of the array.

And you can change those, of course, which is how this works:

>>> nums[2::-1]   # will get indices 2, 1 and 0
[3, 2, 1]

Now herein lies the second important note about slices: When step is negative, the condition that's used to determine whether to take the next element or not is flipped around.

It makes intuitive sense if you think about it for a moment, if we are checking while start < end while also decrementing start at every step, we will never reach the point where the condition becomes false. So we need to flip the condition around to while start > end, in order for slicing to still work.

That explains why nums[-1:-3:-1] returns [6, 5], it's because it starts with the last index, and keeps going until it's decremented till the 3rd last index (which is excluded).

If you want an updated Python code that factors this in, here it is:

def slice(array, start, stop, step=1):
    if start < 0:
        start = len(array) + start
    if stop < 0:
        stop = len(array) + stop

    result = []
    index = start

    if step >= 0:
        while index < stpp:
            result.append(array[index])
            index += step
    else:
        # Negative slice
        while index > stop:
            result.append(array[index])
            index += step

    return result

But what about nums[-1:-3] returning an empty list?

Well that's easy. Since -1 points to the end of the array, and step is 1 (positive), therefore start < end is False from the get go, and the result just stays empty.

Summary

Hopefully it is evident that Python's slices are rather straightforward, once you understand a couple basic concepts about how they function.

Also note, that my slice function isn't an exact implementation of the algorithm, though it comes close. Currently it has no way of not specifying a start or an end, and it also creates an infinite loop for step=0. But apart from that, it's pretty much identical to the Python builtin slice implementation.

So that's pretty much all the math behind Python slices, and how they work under the hood. ✨

What the f-strings?

Tushar Sadhwani — Wed, 07 Jul 2021 21:54:19 +0000

So a couple months ago, I wrote a comprehensive blog on mypy. Today I'm planning to do the same with f-strings, putting everything I know about them into a single source of knowledge. It's an awesome Python feature, and I hope you'll get to learn something new about it ✨

Index

What is an f-string?
Simple examples
A brief history of string formatting
Why f-strings
More examples
- Padding / truncating
- Alignment
- Rounding numbers
- Converting types
- Adding commas to large numbers
Special syntax
- !s, !r and !a
- {x=} syntax
- datetime formatting
Nested formatting
Custom formatting using __format__
Limitations
Conclusion

What is an f-string?

"f-strings" are a feature added to Python in version 3.6, and it's essentially a new, more concise way to embed values inside strings.

They're called f-strings because of the syntax: you put an f before the string literal, like so:

string = f'this is an f-string'

Simple examples

Here's a few basic examples on how to use f-strings to get you started:

>>> name, birth_year = 'Tushar', 2000
>>> print(f'I am {name}, and I was born in {birth_year}')
I am Tushar, and I was born in 2000
>>> import datetime; now = datetime.datetime.now()
>>> print(f'I am {now.year - birth_year} years old')
I am 21 years old
>>> age = now.year - birth_year
>>> print(f'I am {"an adult" if age >= 18 else "a child"}.')
I am an adult.
>>>

A brief history of string formatting

String formatting in Python dates back a long time. The original formatting using the % sign has existed in Python ever since version 1.x (and even before that in C, which is where this feature's inspiration comes from). It allowed you to do most string formatting very easily, even though the syntax was a bit unusual: it uses % flags in your string, which get replaced by the values you pass in. Here's an example:

>>> city = 'London'
>>> print('She lives in %s since %d' % (city, 2000))
She lives in London since 2000
>>>

There's a bunch of these % patterns, and each of them has a meaning:

%s - String
%c - Character (ASCII/Unicode)
%d - Digits (Integer)
%f - Floats
%x - Hexadecimal number, etc.

Each of these can be modified with more information, like padding and alignment, for example:

%9s means a string of length 9.
%-7d means an integer of length 7, but left-aligned.

>>> print('%15f seconds' % 31.415926)
      31.415926 seconds
>>> print('%-15f seconds' % 31.415926)
31.415926       seconds
>>>

Eventually, people realised that the % syntax borrowed from C might not be the most readable way to format strings. So in Python 3.0 (alongside 2.6), A new method was added to the str data type: str.format. Not only was it more obvious in what it was doing, it added a bunch of new features, like dynamic data types, center alignment, index-based formatting, and specifying padding characters. Here's a few examples:

>>> month = 'May'
>>> print('I was born in {}.'.format(month))
I was born in May
>>> blog_title = 'What the f-string?'
>>> print('{title:-^30}'.format(title=blog_title))
------What the f-string?------
>>> print('{:_<15f} seconds'.format(31.415926))
31.415926______ seconds
>>> print('{:>15f} seconds'.format(31.415926))
      31.415926 seconds

Note that if you don't specify an alignment character using <, > or ^, it will always default to left alignment. This is different from %-formatting, as that defaults to right-alignment for numbers.

Along with str.format method, there's also a built-in format function which does the same thing.

For a lot more information about %-formatting and str.format, and all of their (many) features, head to pyformat.info. It is very useful as a reference for the features and syntax.

Why f-strings?

One thing that you should know is that str.format() can already do (almost) everything that f-strings can do. You might be wondering then, "why were f-strings created in the firt place? And why should I care about them?"

Well, it's for two reasons:

it's way more intuitive, and
it's way more readable.

Here's a comparision:

>>> name, age = 'Mark', 31
>>> # str.format version
>>> print('Hi, I'm {0} and I'm {1} years old.'.format(name, age)
Hi, I'm Mark and I'm 31 years old.
>>> # f-string version
>>> print(f'Hi, I'm {name} and I'm {age} years old.')
Hi, I'm Mark and I'm 31 years old.

These two properties, of being intuitive and readable, are part of the core philisophy of Python (refer The Zen of Python).

More Examples

Now, a few examples on everything that you can do with f-strings.

Padding / truncating

>>> pi = 3.141592
>>> print(f'{pi}')
3.141592
>>> print(f'{pi:10}')     # padding to make length 10
  3.141592
>>> print(f'{pi:010}')    # padding with zeroes
003.141592
>>> print(f'{pi:.3}')     # 3 digits total, ignoring decimal
3.14
>>> string = "Hello! this is a string"
>>> print(f'{string:.6}')  # 6 characters
'Hello!'

Alignment

>>> heading = 'Test'
>>> print(f'{heading:>20}')
                Test
>>> print(f'{heading:~>20}')  # specify alignment for custom padding
~~~~~~~~~~~~~~~~Test
>>> print(f'{heading:_<20}')
Test________________
>>> print(f'{heading:=^20}')
========Test========
>>>

Rounding numbers

>>> num = 2.136
>>> print(f'{num}')
2.136
>>> print(f'{num:.3}')  # Rounded up
2.14
>>> print(f'{num:.2}')  # Rounded down
2.1
>>>

Converting types

>>> print(f'{42:c}')       # int to ascii
*
>>> print(f'{604:f}')      # int to float
604.000000
>>> print(f'{84:.2f}%')
84.00%
>>> print(f'{604:x}')      # int to hex
25c
>>> print(f'{604:b}')      # int to binary
1001011100
>>> print(f'{604:0>16b}')  # int to binary, with zero-padding
0000001001011100
>>>

Adding commas to large numbers

>>> large_num = 12_345_678  # int syntax supports underscores :D
>>> print(f'{large_num}')
12345678
>>> print(f'{large_num:,}')
12,345,678

Special syntax

There's a few special set of syntaxes that don't exist in the original %-formatting:

!s !r and !a

>>> from datetime import datetime
>>> datetime.now()                # repr value
datetime.datetime(2021, 7, 6, 2, 21, 56, 698285)
>>> print(datetime.now())         # str value
2021-07-06 02:22:02.772826     
>>> print(f'{datetime.now()}')    # str value by default
2021-07-06 02:22:14.357562
>>> print(f'{datetime.now()!r}')  # repr value
datetime.datetime(2021, 7, 6, 2, 22, 17, 709937)
>>> print(f'{datetime.now()!s}')  # str value
2021-07-06 02:22:20.081837
>>> sparkles = '✨'
>>> print(f'{sparkles}')
✨
>>> print(f'{sparkles!a}')        # ascii-safe value
'\u2728'

`datetime` formatting

>>> from datetime import datetime
>>> print(f'{datetime.now():%A, %B %d %Y}')
Tuesday, July 06 2021

For more info on the % codes specifically for datetime objects, check out the documentation for datetime.strftime, or check out strftime.org.

`{x=}` syntax

This is a new syntax that was added to Python 3.8, and it helps you quickly print out a variable's value, usually for debugging purposes. Essentially, f'{abc=}' is the same as f'abc={abc!r}'.

>>> x, y = 3, 5
>>> print(f'{x=}')
x=3
>>> print(f'{x = }')
x = 3
>>> x, y = y, x+y
>>> print(f'{x=} {y=}')
x=5 y=8
>>> string = 'test'.center(10, '*')
>>> print(f'{string = }')
string = '***test***'
>>> print(f'{10 * 2 = }')
10 * 2 = 20
>>>

Nested formatting

Since we can sometimes have rather complicated format specifications, like ~^30s or 0>12,.2f, it makes sense to be able to extract those out into variables as well. And that's exactly what nested formatting lets us do.

Some things that you can do with

Variable length padding

>>> string = 'Python'
>>> size = 20
>>> print(f'{string:{size}}')
~~~~~~~Python~~~~~~~
>>>

putting in the format spec as a variable, or as multiple variables:

from math import pi

digits = int(input('Enter number of digits of pi: '))
length = int(input('Enter string length: '))
alignment = input('Enter alignment (<, > or ^): ')
padding_char = input('Enter padding character: ')
print(f'{pi:{padding_char}{alignment}{length}.{digits}}')

Output:

Enter number of digits of pi: 8   
Enter string length: 30
Enter alignment (<, > or ^): ^
Enter padding character: _
__________3.1415927___________

Custom formatting using `format`

You can define custom format handling in your own objects by overriding the __format__ method on the formatter object. This allows you to define (mostly) arbitrary formatting semantics.

Here's an example, with more sensible names for datetime formatting:

from datetime import datetime

class BetterDatetime(datetime):
    substitutions = {
        '%day': '%A',
        '%date': '%d',
        '%monthname': '%B',
        '%month': '%m',
        '%year': '%Y'
    }

    def __format__(self, format_spec):
        for token, replacement in self.substitutions.items():
            format_spec = format_spec.replace(token, replacement)

        return super().__format__(format_spec)

print(f'Today is {BetterDatetime.now():%day, %date %monthname %year}')

# Output: Today is Monday, 8 July 2021

Limitations

As amazing as f-strings are, they're not the be-all and end-all of string formatting in Python. (but they come very close!)

Some nitpicks from my side about f-strings are:

You can't separate the string template from the data being embedded. In str.format, you can store the strings themselves as templates in a separate file, like text = '{user} has left'. Then, you can import text and use text.format(user=...). You can't do this with f-strings.
f-strings only work in Python 3.6+, and {x=} syntax only works in Python 3.8+
There's also no real replacement for the str.format_map method using f-strings.

Conclusion

All in all, f-strings are awesome, and everyone should use them :P

Anywho, if you have any questions or suggestions, drop them below. I'd love to hear from you ✨

Ace your LeetCode preparations

Tushar Sadhwani — Mon, 24 May 2021 12:14:56 +0000

I've been practicing my Data Structures and Algorithms on LeetCode for a few months now, and it's an awesome platform. The quality of the questions is generally great, there's very nice explanations for most of the solutions, and there's a very motivated and active community around all of it. Overall, LeetCode is a great platform.

BUT, I had two slight problems with it:

I personally never understood the idea of writing your code and testing it on a web editor. I mean, I have my own programming environment tailored to exactly how I like to write and test my code. And yes, granted that I'll have to use a google doc or something for my interviews, but I still prefer to have control over how I practice.
I'd like to keep and test my solutions locally, in a version-controlled manner, so that I can come back to them later, search through them easily, make changes over time, and to just make sure I never lose my code.

So for this, I created my own LeetCode workflow and local testing library:

python-leetcode-runner

And it has made me many times more productive in solving leetcode problems.

An example workflow

I start with opening my github repo where I store all of my leetcode problems:
Then I head over to any leetcode problem page (example: Plus One). I copy the problem title and format it to be used as my file name (I specifically created the snekify cli tool for that), and then copy over the Solution code snippet.

This creates the filename I want.
I also use static type checking, but feel free to remove that!
Then I copy over the given example test cases into a variable called tests:
Then it's time to write a solution and test it using the pyleet command given by my testing library:
Uh oh. Looks like I missed an edge case. Let's fix the code:

And now all tests pass. Now you can copy your code file (yes, with the tests and all) straight into leetcode's editor and submit it. No more silly "Wrong Answers".

Advanced use cases

Another example: Linked List cycle

Now this has an issue: We need to transform the given input into the linked list data structure, before running our solution.

In other questions for example, you don't just have to match expected output with function output. Like sometimes it might ask you to modify a list in-place, or some might have answers where the order of the output doesn't matter, etc.

For that case, you can provide your own custom validator function.

A validator is a function that receives 3 arguments:

method: your leetcode solution function
inputs: your test inputs tuple
expected: your expected test output value

Validator solution example

Head to the question.
Copy the problem title, format it, and paste the sample code and examples.
Now to be able to run the tests locally, we need to write our own code to convert the array into a linked list:
Now we can solve the question:
Running the tests:

And sure enough, we passed all the test cases. Now simply copy the entire code over to leetcode, and:

You can find the complete solution here.

Extra tips

I use mypy to run static type checking on my code, which ensures stuff like no runtime Null Pointer Exceptions.
I've created a bunch of code snippets that auto-fill the test cases, the validator function, and the assert statements. These snippets can be used in VSCode as of now. More info on the github page.
My solutions to all of the problems are stored in this github repository.

Footnotes

Currently this package only works with Python solutions. But if required, it can be extended to use any language at all. If you're interested in working on other language support, do let me know.

Thanks for reading, I hope this helps you be more productive. ✨

I fixed Google's scripting tool

Tushar Sadhwani — Sat, 08 May 2021 14:45:53 +0000

So someone at Google released a tool called zx a couple days ago, and...

Well, take a look at it for yourself:

I can't be the only one who doesn't like:

The idea of having to install nodejs and npm on my servers just to run a shell script.
The await $ syntax used for every. single. command.

And I thought I could do better, so I tried. And this is what I came up with:

Introducing zxpy

#! /usr/bin/env zxpy
~'echo Hello world!'

file_count = ~'ls -1 | wc -l'
print("file count is:", file_count)

It fixes the two problems I had in the following ways:

It uses Python instead. Not only is Python a much more popular choice for scripting on desktop, it is also pre-installed on a lot more systems. Most linux and MacOS desktops, and arch/ubuntu based linux servers ship with it installed (probably many others do as well!)
It uses a simple ~'shell command' syntax. It's synchronous by default, but in my experience, most shell scripts are as well. It is possible to add async support, and if needed I'll add it in the future, but synchronous as default seems a lot more sane to me. :)

So, why not give it a try?

It's available on pip, and the source code, along with a few examples, is available on GitHub.

The Comprehensive Guide to mypy

Tushar Sadhwani — Wed, 05 May 2021 22:04:17 +0000

Read the original on my personal blog

Mypy is a static type checker for Python. It acts as a linter, that allows you to write statically typed code, and verify the soundness of your types.

All mypy does is check your type hints. It's not like TypeScript, which needs to be compiled before it can work. All mypy code is valid Python, no compiler needed.

This gives us the advantage of having types, as you can know for certain that there is no type-mismatch in your code, just as you can in typed, compiled languages like C++ and Java, but you also get the benefit of being Python 🐍✨ (you also get other benefits like null safety!)

For a more detailed explanation on what are types useful for, head over to the blog I wrote previously: Does Python need types?

This article is going to be a deep dive for anyone who wants to learn about mypy, and all of its capabilities.

If you haven't noticed the article length, this is going to be long. So grab a cup of your favorite beverage, and let's get straight into it.

Index

Setting up mypy
- Using mypy in the terminal
- Using mypy in VSCode
Primitive types
Collection types
Type debugging - Part 1
Union and Optional
Any type
Miscellaneous types
- Tuple
- TypedDict
- Literal
- Final
- NoReturn
Typing classes
Typing namedtuples
Typing decorators
Typing generators
Typing *args and **kwargs
Duck types
Function overloading with @overload
Type type
Typing pre-existing projects
Type debugging - Part 2
Typing context managers
Typing async functions
Making your own Generics
- Generic functions
- Generic classes
- Generic types
Advanced/Recursive type checking with Protocol
Further learning

Setting up mypy

All you really need to do to set it up is pip install mypy.

Using mypy in the terminal

Let's create a regular python file, and call it test.py:

This doesn't have any type definitions yet, but let's run mypy over it to see what it says.



$ mypy test.py
Success: no issues found in 1 source file

🤨

Don't worry though, it's nothing unexpected. As explained in my previous article, mypy doesn't force you to add types to your code. But, if it finds types, it will evaluate them.

This can definitely lead to mypy missing entire parts of your code just because you accidentally forgot to add types.

Thankfully, there's ways to customise mypy to tell it to always check for stuff:



$ mypy --disallow-untyped-defs test.py
test.py:1: error: Function is missing a return type annotation
Found 1 error in 1 file (checked 1 source file)

And now it gave us the error we wanted.

There are a lot of these --disallow- arguments that we should be using if we are starting a new project to prevent such mishaps, but mypy gives us an extra powerful one that does it all: --strict



$ mypy --strict test.py               
test.py:1: error: Function is missing a return type annotation
test.py:4: error: Call to untyped function "give_number" in typed context
Found 2 errors in 1 file (checked 1 source file)

The actual mypy output is all nice and colourful

This gave us even more information: the fact that we're using give_number in our code, which doesn't have a defined return type, so that piece of code also can have unintended issues.

TL;DR: for starters, use mypy --strict filename.py

Using mypy in VSCode

VSCode has pretty good integration with mypy. All you need to get mypy working with it is to add this to your settings.json:



...
  "python.linting.mypyEnabled": true,
  "python.linting.mypyArgs": [
    "--ignore-missing-imports",
    "--follow-imports=silent",
    "--show-column-numbers",
    "--strict"
  ],
...

Now opening your code folder in python should show you the exact same errors in the "Problems" pane:

Also, if you're using VSCode I'll highly suggest installing Pylance from the Extensions panel, it'll help a lot with tab-completion and getting better insight into your types.

Okay, now on to actually fixing these issues.

Primitive types

The most fundamental types that exist in mypy are the primitive types. To name a few:

int
str
float
bool ...

Notice a pattern?

Yup. These are the same exact primitive Python data types that you're familiar with.

And these are actually all we need to fix our errors:

All we've changed is the function's definition in def:

Notice the highlighted part

What this says is "function double takes an argument n which is an int, and the function returns an int.

And running mypy now:



$ mypy --strict test.py
Success: no issues found in 1 source file

Congratulations, you've just written your first type-checked Python program 🎉

We can run the code to verify that it indeed, does work:



$ python test.py
42

I should clarify, that mypy does all of its type checking without ever running the code. It is what's called a static analysis tool (this static is different from the static in "static typing"), and essentially what it means is that it works not by running your python code, but by evaluating your program's structure. What this means is, if your program does interesting things like making API calls, or deleting files on your system, you can still run mypy over your files and it will have no real-world effect.

What is interesting to note, is that we have declared num in the program as well, but we never told mypy what type it is going to be, and yet it still worked just fine.

We could tell mypy what type it is, like so:

And mypy would be equally happy with this as well. But we don't have to provide this type, because mypy knows its type already. Because double is only supposed to return an int, mypy inferred it:

Basic inference actually works without Pylance too.

And inference is cool. For 80% of the cases, you'll only be writing types for function and method definitions, as we did in the first example. One notable exception to this is "empty collection types", which we will discuss now.

Collection types

Collection types are how you're able to add types to collections, such as "a list of strings", or "a dictionary with string keys and boolean values", and so on.

Some collection types include:

List
Dict
Set
DefaultDict
Deque
Counter

Now these might sound very familiar, these aren't the same as the builtin collection types (more on that later).

These are all defined in the typing module that comes built-in with Python, and there's one thing that all of these have in common: they're generic.

I have an entire section dedicated to generics below, but what it boils down to is that "with generic types, you can pass types inside other types". Here's how you'd use collection types:

This tells mypy that nums should be a list of integers (List[int]), and that average returns a float.

Here's a couple more examples:

Remember when I said that empty collections is one of the rare cases that need to be typed? This is because there's no way for mypy to infer the types in that case:

Since the set has no items to begin with, mypy can't statically infer what type it should be.

PS:

Small note, if you try to run mypy on the piece of code above, it'll actually succeed. It's because the mypy devs are smart, and they added simple cases of look-ahead inference. Meaning, new versions of mypy can figure out such types in simple cases. Keep in mind that it doesn't always work.

To fix this, you can manually add in the required type:

Note: Starting from Python 3.7, you can add a future import, from __future__ import annotations at the top of your files, which will allow you to use the builtin types as generics, i.e. you can use list[int] instead of List[int]. If you're using Python 3.9 or above, you can use this syntax without needing the __future__ import at all. However, there are some edge cases where it might not work, so in the meantime I'll suggest using the typing.List variants. This is detailed in PEP 585.

Type debugging - Part 1

Let's say you're reading someone else's — or your own past self's — code, and it's not really apparent what the type of a variable is. The code is using a lot of inference, and it's using some builtin methods that you don't exactly remember how they work, bla bla.

Thankfully mypy lets you reveal the type of any variable by using reveal_type:

Running mypy on this piece of code gives us:



$ mypy --strict test.py 
test.py:12: note: Revealed type is 'builtins.int'

Ignore the builtins for now, it's able to tell us that counts here is an int.

Cool, right? You don't need to rely on an IDE or VSCode, to use hover to check the types of a variable. A simple terminal and mypy is all you need. (although VSCode internally uses a similar process to this to get all type informations)

However, some of you might be wondering where reveal_type came from. We didn't import it from typing... is it a new builtin? Is that even valid in python?

And sure enough, if you try to run the code:



py test.py    
Traceback (most recent call last):
  File "/home/tushar/code/test/test.py", line 12, in <module>
    reveal_type(counts)
NameError: name 'reveal_type' is not defined

reveal_type is a special "mypy function". Since python doesn't know about types (type annotations are ignored at runtime), only mypy knows about the types of variables when it runs its type checking. So, only mypy can work with reveal_type.

All this means, is that you should only use reveal_type to debug your code, and remove it when you're done debugging.

Union and Optional

So far, we have only seen variables and collections that can hold only one type of value. But what about this piece of code?

What's the type of fav_color in this code?

Let's try to do a reveal_type:

BTW, since this function has no return statement, its return type is None.

Running mypy on this:



$ mypy test.py 
test.py:5: note: Revealed type is 'Union[builtins.str*, None]'

And we get one of our two new types: Union. Specifically, Union[str, None].

All this means, is that fav_color can be one of two different types, either str, or None.

And unions are actually very important for Python, because of how Python does polymorphism. Here's a simpler example:



$ python test.py
Hi!
This is a test
of polymorphism

Now let's add types to it, and learn some things by using our friend reveal_type:

Can you guess the output of the reveal_types?



$ mypy test.py 
test.py:4: note: Revealed type is 'Union[builtins.str, builtins.list[builtins.str]]'
test.py:8: note: Revealed type is 'builtins.list[builtins.str]'
test.py:11: note: Revealed type is 'builtins.str'

Mypy is smart enough, where if you add an isinstance(...) check to a variable, it will correctly assume that the type inside that block is narrowed to that type.

In our case, item was correctly identified as List[str] inside the isinstance block, and str in the else block.

This is an extremely powerful feature of mypy, called Type narrowing.

Now, here's a more contrived example, a tpye-annotated Python implementation of the builtin function abs:

And that's everything you need to know about Union.

... so what's Optional you ask?

Well, Union[X, None] seemed to occur so commonly in Python, that they decided it needs a shorthand. Optional[str] is just a shorter way to write Union[str, None].

Any type

If you ever try to run reveal_type inside an untyped function, this is what happens:



$ mypy test.py         
test.py:6: note: Revealed type is 'Any'
test.py:6: note: 'reveal_type' always outputs 'Any' in unchecked functions

Didn't use --strict, or it'd throw errors

The revealed type is told to be Any.

Any just means that anything can be passed here. Whatever is passed, mypy should just accept it. In other words, Any turns off type checking.

Of course, this means that if you want to take advantage of mypy, you should avoid using Any as much as you can.

But since Python is inherently a dynamically typed language, in some cases it's impossible for you to know what the type of something is going to be. For such cases, you can use Any. For example:

You can also use Any as a placeholder value for something while you figure out what it should be, to make mypy happy in the meanwhile. But make sure to get rid of the Any if you can .

Miscellaneous types

Tuple

You might think of tuples as an immutable list, but Python thinks of it in a very different way.

Tuples are different from other collections, as they are essentially a way to represent a collection of data points related to an entity, kinda similar to how a C struct is stored in memory. While other collections usually represent a bunch of objects, tuples usually represent a single object.

A good example is sqlite:

Tuples also come in handy when you want to return multiple values from a function, for example:

Because of these reasons, tuples tend to have a fixed length, with each index having a specific type. (Our sqlite example had an array of length 3 and types int, str and int respectively.

Here's how you'd type a tuple:

However, sometimes you do have to create variable length tuples. You can use the Tuple[X, ...] syntax for that.

The ... in this case simply means there's a variable number of elements in the array, but their type is X. For example:

TypedDict

A TypedDict is a dictionary whose keys are always string, and values are of the specified type. At runtime, it behaves exactly like a normal dictionary.

By default, all keys must be present in a TypedDict. It is possible to override this by specifying total=False.

Literal

A Literal represents the type of a literal value. You can use it to constrain already existing types like str and int, to just some specific values of them. Like so:



$ mypy test.py 
test.py:7: error: Argument 1 to "i_only_take_5" has incompatible type "Literal[6]"; expected "Literal[5]"

This has some interesting use-cases. A notable one is to use it in place of simple enums:



$ mypy test.py
test.py:8: error: Argument 1 to "make_request" has incompatible type "Literal['DLETE']"; expected "Union[Literal['GET'], Literal['POST'], Literal['DELETE']]"

Oops, you made a typo in 'DELETE'! Don't worry, mypy saved you an hour of debugging.

Final

Final is an annotation that declares a variable as final. What that means that the variable cannot be re-assigned to. This is similar to final in Java and const in JavaScript.

NoReturn

NoReturn is an interesting type. It's rarely ever used, but it still needs to exist, for that one time where you might have to use it.

There are cases where you can have a function that might never return. Two possible reasons that I can think of for this are:

The function always raises an exception, or
The function is an infinite loop.

Here's an example of both:

Note that in both these cases, typing the function as -> None will also work. But if you intend for a function to never return anything, you should type it as NoReturn, because then mypy will show an error if the function were to ever have a condition where it does return.

For example, if you edit while True: to be while False: or while some_condition() in the first example, mypy will throw an error:



$ mypy test.py 
test.py:6: error: Implicit return in function which does not return

Typing classes

All class methods are essentially typed just like regular functions, except for self, which is left untyped. Here's a simple Stack class:

If you've never seen the {x!r} syntax inside f-strings, it's a way to use the repr() of a value. For more information, pyformat.info is a very good resource for learning Python's string formatting features.

There's however, one caveat to typing classes: You can't normally access the class itself inside the class' function declarations (because the class hasn't been finished declaring itself yet, because you're still declaring its methods).

So something like this isn't valid Python:



$ mypy --strict test.py
Success: no issues found in 1 source file

$ python test.py
Traceback (most recent call last):
  File "/home/tushar/code/test/test.py", line 11, in <module>
    class MyClass:
  File "/home/tushar/code/test/test.py", line 15, in MyClass
    def copy(self) -> MyClass:
NameError: name 'MyClass' is not defined

There's two ways to fix this:

Turn the classname into a string: The creators of PEP 484 and Mypy knew that such cases exist where you might need to define a return type which doesn't exist yet. So, mypy is able to check types if they're wrapped in strings.

Use from __future__ import annotations. What this does, is turn on a new feature in Python called "postponed evaluation of type annotations". This essentially makes Python treat all type annotations as strings, storing them in the internal __annotations__ attribute. Details are described in PEP 563.

Starting with Python 3.11, the Postponed evaluation behaviour will become default, and you won't need to have the __future__ import anymore.

Typing namedtuples

namedtuples are a lot like tuples, except every index of their fields is named, and they have some syntactic sugar which allow you to access its properties like attributes on an object:

Since the underlying data structure is a tuple, and there's no real way to provide any type information to namedtuples, by default this will have a type of Tuple[Any, Any, Any].

To combat this, Python has added a NamedTuple class which you can extend to have the typed equivalent of the same:

Inner workings of NamedTuple:

If you're curious how NamedTuple works under the hood: age: int is a type declaration, without any assignment (like age : int = 5).

Type declarations inside a function or class don't actually define the variable, but they add the type annotation to that function or class' metadata, in the form of a dictionary entry, into x.__annotations__.

Doing print(ishan.__annotations__) in the code above gives us {'name': <class 'str'>, 'age': <class 'int'>, 'bio': <class 'str'>}.

typing.NamedTuple uses these annotations to create the required tuple.

Typing decorators

Decorators are a fairly advanced, but really powerful feature of Python. If you don't know anything about decorators, I'd recommend you to watch Anthony explains decorators, but I'll explain it in brief here as well.

A decorator is essentially a function that wraps another function. Decorators can extend the functionalities of pre-existing functions, by running other side-effects whenever the original function is called. A decorator decorates a function by adding new functionality.

A simple example would be to monitor how long a function takes to run:

To be able to type this, we'd need a way to be able to define the type of a function. That way is called Callable.

Callable is a generic type with the following syntax:

Callable[[<list of argument types>], <return type>]

The types of a function's arguments goes into the first list inside Callable, and the return type follows after. A few examples:

Here's how you'd implenent the previously-shown time_it decorator:

Note: Callable is what's called a Duck Type. What it means, is that you can create your own custom object, and make it a valid Callable, by implementing the magic method called __call__. I have a dedicated section where I go in-depth about duck types ahead.

Typing generators

Generators are also a fairly advanced topic to completely cover in this article, and you can watch
Anthony explains generators if you've never heard of them. A brief explanation is this:

Generators are a bit like perpetual functions. Instead of returning a value a single time, they yield values out of them, which you can iterate over. When you yield a value from an iterator, its execution pauses. But when another value is requested from the generator, it resumes execution from where it was last paused. When the generator function returns, the iterator stops.

Here's an example:

To add type annotations to generators, you need typing.Generator. The syntax is as follows:

Generator[yield_type, throw_type, return_type]

With that knowledge, typing this is fairly straightforward:

Since we're not raising any errors in the generator, throw_type is None. And although the return type is int which is correct, we're not really using the returned value anyway, so you could use Generator[str, None, None] as well, and skip the return part altogether.

Typing `*args` and `**kwargs`

*args and **kwargs is a feature of python that lets you pass any number of arguments and keyword arguments to a function (that's what the name args and kwargs stands for, but these names are just convention, you can name the variables anything). Anthony explains args and kwargs

All the extra arguments passed to *args get turned into a tuple, and kewyord arguments turn into a dictionay, with the keys being the string keywords:

Since the *args will always be of typle Tuple[X], and **kwargs will always be of type Dict[str, X], we only need to provide one type value X to type them. Here's a practical example:

Duck types

Duck types are a pretty fundamental concept of python: the entirety of the Python object model is built around the idea of duck types.

Quoting Alex Martelli:

"You don't really care for IS-A -- you really only care for BEHAVES-LIKE-A-(in-this-specific-context), so, if you do test, this behaviour is what you should be testing for."

What it means is that Python doesn't really care what the type of an object is, but rather how does it behave.

I had a short note above in typing decorators that mentioned duck typing a function with __call__, now here's the actual implementation:

PS.

> Running mypy over the above code is going to give a cryptic error about "Special Forms", don't worry about that right now, we'll fix this in the Protocol section. All I'm showing right now is that the Python code works.

You can see that Python agrees that both of these functions are "Call-able", i.e. you can call them using the x() syntax. (this is why the type is called Callable, and not something like Function)

What duck types provide you is to be able to define your function parameters and return types not in terms of concrete classes, but in terms of how your object behaves, giving you a lot more flexibility in what kinds of things you can utilize in your code now, and also allows much easier extensibility in the future without making "breaking changes".

A simple example here:

Running this code with Python works just fine. But running mypy over this gives us the following error:



$ mypy test.py 
test.py:12: error: Argument 1 to "count_non_empty_strings" has incompatible type "ValuesView[str]"; expected "List[str]"

ValuesView is the type when you do dict.values(), and although you could imagine it as a list of strings in this case, it's not exactly the type List.

In fact, none of the other sequence types like tuple or set are going to work with this code. You could patch it for some of the builtin types by doing strings: Union[List[str], Set[str], ...] and so on, but just how many types will you add? And what about third party/custom types?

The correct solution here is to use a Duck Type (yes, we finally got to the point). The only thing we want to ensure in this case is that the object can be iterated upon (which in Python terms means that it implements the __iter__ magic method), and the right type for that is Iterable:

And now mypy is happy with our code.

There are many, many of these duck types that ship within Python's typing module, and a few of them include:

Sequence for defining things that can be indexed and reversed, like List and Tuple.
MutableMapping, for when you have a key-value pair kind-of data structure, like dict, but also others like defaultdict, OrderedDict and Counter from the collections module.
Collection, if all you care about is having a finite number of items in your data structure, eg. a set, list, dict, or anything from the collections module.

If you haven't already at this point, you should really look into how python's syntax and top level functions hook into Python's object model via __magic_methods__, for essentially all of Python's behaviour. The documentation for it is right here, and there's an excellent talk by James Powell that really dives deep into this concept in the beginning.

Function overloading with `@overload`

Let's write a simple add function that supports int's and float's:

The implementation seems perfectly fine... but mypy isn't happy with it:



$ test.py:15: error: No overload variant of "__getitem__" of "list" matches argument type "float"
test.py:15: note: Possible overload variants:
test.py:15: note:     def __getitem__(self, int) -> int
test.py:15: note:     def __getitem__(self, slice) -> List[int]

What mypy is trying to tell us here, is that in the line:

print(joined_list[last_index])

last_index could be of type float. And checking with reveal_type, that definitely is the case:

And since it could, mypy won't allow you to use a possible float value to index a list, because that will error out.

One thing we could do is do an isinstance assertion on our side to convince mypy:

But this will be pretty cumbersome to do at every single place in our code where we use add with int's. Also we as programmers know, that passing two int's will only ever return an int. But how do we tell mypy that?

Answer: use @overload. The syntax basically replicates what we wanted to say in the paragraph above:

And now mypy knows that add(3, 4) returns an int.

Note that Python has no way to ensure that the code actually always returns an int when it gets int values. It's your job as the programmer providing these overloads, to verify that they are correct. This is why in some cases, using assert isinstance(...) could be better than doing this, but for most cases @overload works fine.

Also, in the overload definitions -> int: ..., the ... at the end is a convention for when you provide type stubs for functions and classes, but you could technically write anything as the function body: pass, 42, etc. It'll be ignored either way.

Another good overload example is this:

`Type` type

Type is a type used to type classes. It derives from python's way of determining the type of an object at runtime:

You'd usually use issubclass(x, int) instead of type(x) == int to check for behaviour, but sometimes knowing the exact type can help, for eg. in optimizations.

Since type(x) returns the class of x, the type of a class C is Type[C]:

We had to use Any in 3 places here, and 2 of them can be eliminated by using generics, and we'll talk about it later on.

Typing pre-existing projects

If you need it, mypy gives you the ability to add types to your project without ever modifying the original source code. It's done using what's called "stub files".

Stub files are python-like files, that only contain type-checked variable, function, and class definitions. It's kindof like a mypy header file.

You can make your own type stubs by creating a .pyi file:

Now, run mypy on the current folder (make sure you have an __init__.py file in the folder, if not, create an empty one).



$ ls
__init__.py  test.py  test.pyi

$ mypy --strict .
Success: no issues found in 2 source files

Type debugging - Part 2

Since we are on the topic of projects and folders, let's discuss another one of pitfalls that you can find yourselves in when using mypy.

The first one is PEP 420

A fact that took me some time to realise, was that for mypy to be able to type-check a folder, the folder must be a module.

Let's say you find yourself in this situatiion:



$ tree
.
├── test.py
└── utils
    └── foo.py

1 directory, 2 files

$ cat test.py               
from utils.foo import average

print(average(3, 4))

$ cat utils/foo.py 
def average(x: int, y: int) -> float:
    return float(x + y) / 2

$ py test.py      
3.5

$ mypy test.py 
test.py:1: error: Cannot find implementation or library stub for module named 'utils.foo'
test.py:1: note: See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports
Found 1 error in 1 file (checked 1 source file)

What's the problem? Python is able to find utils.foo no problems, why can't mypy?

The error is very cryptic, but the thing to focus on is the word "module" in the error. utils.foo should be a module, and for that, the utils folder should have an __init__.py, even if it's empty.



$ tree              
.
├── test.py
└── utils
    ├── foo.py
    └── __init__.py

1 directory, 3 files

$ mypy test.py
Success: no issues found in 1 source file

Now, the same issue re-appears if you're installing your package via pip, because of a completely different reason:



$ tree ..
..
├── setup.py
├── src
│   └── mypackage
│       ├── __init__.py
│       └── utils
│           ├── foo.py
│           └── __init__.py
└── test
    └── test.py

4 directories, 5 files

$ cat ../setup.py
from setuptools import setup, find_packages

setup(
    name="mypackage",
    packages = find_packages('src'),
    package_dir = {"":"src"}
)

$ pip install ..
[...]
successfully installed mypackage-0.0.0

$ cat test.py
from mypackage.utils.foo import average

print(average(3, 4))

$ python test.py
3.5

$ mypy test.py
test.py:1: error: Cannot find implementation or library stub for module named 'mypackage.utils.foo'
test.py:1: note: See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports
Found 1 error in 1 file (checked 1 source file)

What now? Every folder has an __init__.py, it's even installed as a pip package and the code runs, so we know that the module structure is right. What gives?

Well, turns out that pip packages aren't type checked by mypy by default. This behaviour exists because type definitions are opt-in by default. Python packages aren't expected to be type-checked, because mypy types are completely optional. If mypy were to assume every package has type hints, it would show possibly dozens of errors because a package doesn't have proper types, or used type hints for something else, etc.

To opt-in for type checking your package, you need to add an empty py.typed file into your package's root directory, and also include it as metadata in your setup.py:



$ tree ..
..
├── setup.py
├── src
│   └── mypackage
│       ├── __init__.py
│       ├── py.typed
│       └── utils
│           ├── foo.py
│           └── __init__.py
└── test
    └── test.py

4 directories, 6 files

$ cat ../setup.py
from setuptools import setup, find_packages

setup(
    name="mypackage",
    packages = find_packages(
        where = 'src',
    ),
    package_dir = {"":"src"},
    package_data={
        "mypackage": ["py.typed"],
    }
)

$ mypy test.py
Success: no issues found in 1 source file

There's yet another third pitfall that you might encounter sometimes, which is if a.py declares a class MyClass, and it imports stuff from a file b.py which requires to import MyClass from a.py for type-checking purposes.

This creates an import cycle, and Python gives you an ImportError. To avoid this, simple add an if typing.TYPE_CHECKING: block to the import statement in b.py, since it only needs MyClass for type checking. Also, everywhere you use MyClass, add quotes: 'MyClass' so that Python is happy.

Typing Context managers

Context managers are a way of adding common setup and teardown logic to parts of your code, things like opening and closing database connections, establishing a websocket, and so on. On the surface it might seem simple but it's a pretty extensive topic, and if you've never heard of it before, Anthony covers it here.

To define a context manager, you need to provide two magic methods in your class, namely __enter__ and __exit__. They're then called automatically at the start and end if your with block.

You might have used a context manager before: with open(filename) as file: - this uses a context manager underneath. Speaking of which, let's write our own implementation of open:

Typing async functions

The typing module has a duck type for all types that can be awaited: Awaitable.

Just like how a regular function is a Callable, an async function is a Callable that returns an Awaitable:

Generics

Generics (or generic types) is a language feature that lets you "pass types inside other types".

I personally think it is best explained with an example:

Let's say you have a function that returns the first item in an array. To define this, we need this behaviour:

"Given a list of type List[X], we will be returning an item of type X."

And that's exactly what generic types are: defining your return type based on the input type.

Generic functions

We've seen make_object from the Type type section before, but we had to use Any to be able to support returning any kind of object that got created by calling cls(*args). But, we don't actually have to do that, because we can use generics. Here's how you'd do that:

T = TypeVar('T') is how you declare a generic type in Python. What the function definition now says, is "If i give you a class that makes T's, you'll be returning an object T".

And sure enough, the reveal_type on the bottom shows that mypy knows c is an object of MyClass.

The generic type name T is another convention, you can call it anything.

Another example: largest, which returns the largest item in a list:

This seems good, but mypy isn't happy:



$ mypy --strict test.py
test.py:10: error: Unsupported left operand type for > ("T")
Found 1 error in 1 file (checked 1 source file)

This is because you need to ensure you can do a < b on the objects, to compare them with each other, which isn't always the case:



>>> {} < {}
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: '<' not supported between instances of 'dict' and 'dict'

For this, we need a Duck Type that defines this "a less than b" behaviour.

And although currently Python doesn't have one such builtin hankfully, there's a "virtual module" that ships with mypy called _typeshed. It has a lot of extra duck types, along with other mypy-specific features.

Now, mypy will only allow passing lists of objects to this function that can be compared to each other.

If you're wondering why checking for < was enough while our code uses >, that's how python does comparisons. I'm planning to write an article on this later.

Note that _typeshed is not an actual module in Python, so you'll have to import it by checking if TYPE_CHECKING to ensure python doesn't give a ModuleNotFoundError. And since SupportsLessThan won't be defined when Python runs, we had to use it as a string when passed to TypeVar.

At this point you might be interested in how you could implement one of your own such SupportsX types. For that, we have another section below: Protocols.

Generic classes

we implemented a simple Stack class in typing classes, but it only worked for integers. But we can very simply make it work for any type.

To do that, we need mypy to understand what T means inside the class. And for that, we need the class to extend Generic[T], and then provide the concrete type to Stack:

You can pass as many TypeVars to Generic[...] as you need, for eg. to make a generic dictionary, you might use class Dict(Generic[KT, VT]): ...

Generic types

Generic types (a.k.a. Type Aliases) allow you to put a commonly used type in a variable -- and then use that variable as if it were that type.

And mypy lets us do that very easily: with literally just an assignment. The generics parts of the type are automatically inferred.

There is an upcoming syntax that makes it clearer that we're defining a type alias: Vector: TypeAlias = Tuple[int, int]. This is available starting Python 3.10

Just like how we were able to tell the TypeVar T before to only support types that SupportLessThan, we can also do that

AnyStr is a builtin restricted TypeVar, used to define a unifying type for functions that accept str and bytes:

This is different from Union[str, bytes], because AnyStr represents Any one of those two types at a time, and thus doesn't concat doesn't accept the first arg as str and the second as bytes.

Advanced/Recursive type checking with `Protocol`

We implemented FakeFuncs in the duck types section above, and we used isinstance(FakeFuncs, Callable) to verify that the object indeed, was recognized as a callable.

But what if we need to duck-type methods other than __call__?

If we want to do that with an entire class: That becomes harder. Say we want a "duck-typed class", that "has a get method that returns an int", and so on. We don't actually have access to the actual class for some reason, like maybe we're writing helper functions for an API library.

To do that, we need to define a Protocol:

Using this, we were able to type check out code, without ever needing a completed Api implementaton.

This is extremely powerful. We're essentially defining the structure of object we need, instead of what class it is from, or it inherits from. This gives us the flexibility of duck typing, but on the scale of an entire class.

Remember SupportsLessThan? if you check its implementation in _typeshed, this is it:

Yeah, that's the entire implementaton.

What this also allows us to do is define Recursive type definitions. The simplest example would be a Tree:

Note that for this simple example, using Protocol wasn't necessary, as mypy is able to understand simple recursive structures. But for anything more complex than this, like an N-ary tree, you'll need to use Protocol.

Structural subtyping and all of its features are defined extremely well in PEP 544.

Further learning

If you're interested in reading even more about types, mypy has excellent documentation, and you should definitely read it for further learning, especially the section on Generics.

I referenced a lot of Anthony Sottile's videos in this for topics out of reach of this article. He has a YouTube channel where he posts short, and very informative videos about Python.

You can find the source code the typing module here, of all the typing duck types inside the _collections_abc module, and of the extra ones in _typeshed in the typeshed repo.

A topic that I skipped over while talking about TypeVar and generics, is Variance. It's a topic in type theory that defines how subtypes and generics relate to each other. If you want to learn about it in depth, there's documentation in mypy docs of course, and there's two more blogs I found which help grasp the concept, here and here.

A bunch of this material was cross-checked using Python's official documentation, and honestly their docs are always great. Also, the "Quick search" feature works surprisingly well.

There's also quite a few typing PEPs you can read, starting with the kingpin: PEP 484, and the accompanying PEP 526. Other PEPs I've mentioned in the article above are PEP 585, PEP 563, PEP 420 and PEP 544.

And that's it!

I've worked pretty hard on this article, distilling down everything I've learned about mypy in the past year, into a single source of knowledge. If you have any doubts, thoughts, or suggestions, be sure to comment below and I'll get back to you.

Also, if you read the whole article till here, Thank you! And congratulations, you now know almost everything you'll need to be able to write fully typed Python code in the future. I hope you liked it ✨

Does Python need types?

Tushar Sadhwani — Wed, 28 Apr 2021 19:50:15 +0000

I'm a huge fan of Python. It's by far the simplest general purpose language, that you can just pick up and start building amazing things with.

But for the past year or so, I've been working on frontend projects, and I've really enjoyed using Typescript. It's essentially JavaScript, but with fancy features built on top of it like Static Type checking and Null safety, and it was awesome how much it helped in writing robust, bug free code.

So I went out to find if Python has such an equivalent, and sure enough, there was.

It's called mypy, and it is amazing. It works so well in-fact, that I can never go back to writing plain Python now — and this article will be your introduction to it.

But before all that, let's figure out what's the deal with Types.

Why types?

What it essentially means is if you have a type system, every variable has a pre-decided type associated with it.

Untyped vs. typed Python code

Note that, while this might not look like it, the typed version is perfectly valid Python code.

What it also means, is you can't pass values of the wrong type anywhere. The type checker doesn't let you.

Which is extremely valuable if you think about it - I've lost count how many TypeError's I've seen in Python over the years!

Just having the confidence that there's no such place in your code where accidentally passed a str where an int was expected, eliminates an entire class of bugs from your codebase.

Not only that - you get a bunch of other benefits, namely:

Self-documenting code
None-awareness
Better autocompletion and IDE support

I'll go over all these points in detail.

Self-documenting code

Imagine you have this piece of code:

def add_orders(self, orders):
    for order in orders:
        self.pending_ids.add(order.id)

Seems rather simple, doesn't it? We seem to have a list of order's, and we add each order's id to a set called pending_ids.

But what are order's here...

It's hard to tell. In a large codebase, you might have to search pretty hard to find out which part of the code is calling add_orders, and where the data in that is coming from, to eventually find out that it's supposed to be just a namedtuple.

How about this instead:

from models import Order

def add_orders(self, orders: list[Order]) -> None:
    for order in orders:
        self.pending_ids.add(order.id)

Now it's instantly clear, that everywhere add_orders is used, it's going to be exactly of that type.

`None`-awareness

What I mean by this, is that not only can you not pass wrong types of values around, you also can't pass values that could be None, to places that don't expect the value to be possibly None.

Here's an example:

User = namedtuple('User', ['name', 'favorites'])

def fetch_users():
    users = []
    for _ in range(3):
        user_dict = get_user_from_api()
        user = User(
            name=user_dict.get('name', 'Anonymous'),
            favorites=user_dict.get('favorites')
        )
        users.append(user)

    return users

def print_favorite_colors(users):
    for user in users:
        print(user.favorites.get('color'))

users = fetch_users()
print_favorite_colors(users)

... and on first glance, this looks fine. We're using .get so we shouldn't get a KeyError anywhere, so we should be fine, right?

Now here's the typed version of the same code:

class User(NamedTuple):
    name: str
    favorites: Optional[dict[str, str]]

def fetch_users() -> list[User]:
    users = []
    for _ in range(3):
        user_dict = get_user_from_api()
        user = User(
            name=user_dict.get('name', 'Anonymous'),
            favorites=user_dict.get('favorites')
        )
        users.append(user)

    return users


def print_favorite_colors(users: list[User]) -> None:
    for user in users:
        print(user.favorites.get('color'))


users = fetch_users()
print_favorite_colors(users)

And as soon as you add types, you see one error:

You forgot that user.favorites could be None, which would crash your entire application.

Good thing mypy caught it before your clients did.

Better autocompletion and IDE support

This is honestly my favorite part of working with typed Python. The amount of autocompletion static types give me is awesome, and it increases my productivity ten-fold, because I rarely have to open the documentation anymore.

Where can I use it?

Now I can hear you saying, "All of this sounds very cool. But where can I use this mypy-thing in my Python codebase?"

And turns out, you can start gradually adding types to your existing Python codebase, one function and one class at a time. It will infer as much information as it can from the amount of type information it has, and will reduce your bugs no matter how small you start.

Conclusion

So, this was my introduction to you, to the world of static type checking in Python. Are you interested in learning more about it? I'll be dropping a detailed guide to mypy very soon, so stay tuned.

I'd also love to hear your thoughs on this article, so let me know what you think about mypy down in the comments.

Connecting Android Apps to localhost, Simplified

Tushar Sadhwani — Sat, 17 Apr 2021 19:20:08 +0000

P.S. if you're in a hurry, find the correct solution here

I was working on a full stack side project a few months ago, and I wanted to make API requests from my android app to my desktop, and for some reason localhost:8000 wasn't simply accessible by my phone.

Well, understandable, I know that every device's localhost is independent and cannot be accessed by your home network (your Wi-Fi, for example). So the localhost on my laptop won't be able to access the localhost on my phone.

So, I asked Google for help. And I got a large number of solutions, the most sensible one being "use the internal IP address of your PC", et voilà.

The Bad way

Use ipconfig if you're on windows

Running ip addr on my machine tells me that the laptop's internal IP is 192.168.29.76.

And sure enough, as long as both devices were using the same Wi-Fi network, accessing http://192.168.29.76:8000 instead of http://localhost:8000 did work. My Android app can now make web requests to my local backend server 🎉

But this solution is... a bit unstable.

The internal IP of your laptop can keep changing whenever it connects to Wi-Fi, depending on various factors. And everytime it changes, you have to change the URL in your app's code, which is not ideal.

There's other ways as well like using ngrok, but it faces similar issues.

The Correct, easy way

use adb reverse.

Yup, that's it.

Connect your android device to your pc via USB, ensure you have adb setup, and run this in your terminal:



adb reverse tcp:8000 tcp:8000

Now, your mobile can access localhost:8000, just like your PC. (you can replace 8000 with whichever port you want to forward)

Why did nobody tell me this?

Yeah, I was also surprised when I was unable to find anyone on Google or StackOverflow mentioning the existence of adb reverse when I tried to look for it.

Which is why I wrote this blog. Now hopefully, adb reverse will become more popular.

If you know why adb reverse isn't as popular, let me know. Also, if you know another android developer that should know about this little productivity hack, why not share this blog with them? :P

Cover image courtesy of Fotis Fotopoulos on Unsplash

How to setup PostgreSQL and PGAdmin on Manjaro Linux / Arch

Tushar Sadhwani — Sat, 26 Sep 2020 18:48:35 +0000

This guide is here just because I've messed up the installs on arch before, and turns out it's actually pretty easy to do.

Step 1 - Install the dependencies

sudo pacman -S yay
yay postgresql pgadmin4

This should automatically setup your postgres user and group.

Step 2 - Setup postgres service

sudo -u postgres -i # login as postgres
initdb --locale $LANG -E UTF8 -D '/var/lib/postgres/data/'
exit

sudo systemctl enable --now postgresql
sudo systemctl status postgresql # to check for any errors

Step 3 - Setup password

psql -U postgres

postgres=# \password # to set password

Step 4 - Setup connection security

$ su

# cd /var/lib/postgres/data
# cp pg_hba.conf pg_hba.conf.backup # in case you mess up
# nano pg_hba.conf

Your default pg_hba.conf might look like this:

 TYPE  DATABASE        USER            ADDRESS                 METHOD

# "local" is for Unix domain socket connections only
local   all             all                                     trust
# IPv4 local connections:
host    all             all             127.0.0.1/32            trust
# IPv6 local connections:
host    all             all             ::1/128                 trust
# Allow replication connections from localhost, by a user with the
# replication privilege.
local   replication     all                                     trust
host    replication     all             127.0.0.1/32            trust
host    replication     all             ::1/128                 trust

"Method" is set to trust, meaning it won't ask for the password to anyone. To fix that, change the method from trust to md5 everywhere.

And that should be it for postgres!

Bonus: shortcuts

> psql dbname postgres # to directly open a database

postgres=# \c                       # see current database
postgres=# \l                       # see list of databases
postgres=# \c dbname                # set database
postgres=# create database dbname;  # create database
postgres=# \dt                      # see list of tables

Step 6 - PgAdmin

Open up pgadmin, click on "Add New Server", and add the following:

Host: localhost
Port: 5432
Maintenance database: postgres
Username: postgres
Password: <your password>

And PgAdmin should work just fine.

How I made my own URL shortener for free

Tushar Sadhwani — Tue, 01 Sep 2020 13:47:31 +0000

URL shorteners are great. You can take a long URL and make it 6–8 characters, you know the drill.

And the free URL shorteners are pretty nice too! bit.ly is what I used, and it did its job pretty well, but free public shorteners have a few issues:

Finding the perfect short URL is hard: most of the short ones that make sense, are usually already taken.
the URLs are hard to update: (and I don’t know if this is a bit.ly specific issue), but you can’t change the URL the short link points to, once it has been created.

So I decided to make my own instead.

Step 1: Getting a short domain

The entire point of a URL shortener is to be short, and for that you need to get a short domain name. Fortunately, services like Dot tk exist, which provide domains from various TLDs at no cost. (If you’re a university student, you can also use the GitHub Student pack for domains like .me).

I got the domain tshr.me for myself.

free short domains for everyone!

Step 2: Making the URLs redirect

Traditionally, URLs run off of a hosted web server, which are also relatively easy to make, as shown in this video for example.

But the issue with a web server solution is that you have to host it, and hosting ain’t free.

Sure, there’s services like Heroku, which is a great platform, but it’s inadequate for our use case:

Its free dynos only run for 550 hours a month, which means you either have to add your credit card information, or there will be no guarantees that your links will always work.
Free dynos go to sleep after 1 hour of inactivity, meaning if someone tries to access your URLs after a long time, they might have to wait a significantly long time before they get redirected. (This can possibly be fixed by scheduling cron jobs to ping the server, but it’s not a great solution.)
And most importantly: Heroku’s free tier doesn’t support custom domain names!

So we will have to use a different approach.

GitHub to the rescue — Part 1

You should be knowing already, that GitHub can hosts static websites for free on its platform called GitHub Pages.

But you might be thinking, “Yes, I know you can use GitHub for static websites, but you can’t use it to host web servers, dummy.”

… or can you. 🤨

Nah, you can’t host a web server on GitHub, but you can do the next best thing: host a static URL shortener.

This script, when ran in a browser, will redirect you to medium.com immediately. Put this file in a repository on GitHub, add the domain name tshr.me to the GitHub Pages custom domain field, and BOOM!💥

tshr.me now redirects to medium.com.

… you don’t seem very impressed. Don’t worry, we’ll get there.

Just Automate it (with Python!)

Here comes the fun part:

If we could generate all the static short links we want that use the above technique to redirect to the actual URL we want, then this technique might be of some actual use, right?

So I wrote a python scripts that takes in JSON data of various short links and their corresponding redirect URLs, and generates all the HTML pages for me.

The corresponding JSON data looks like this:

Now we’re talking! Running this script generates your HTML redirects and places them neatly in a folder, which you can upload to your GitHub.

But this isn’t exactly the most convenient, right? Every time you have to update your URLs, you have to update the JSON file, run the python code, push the folder to GitHub…

It just sounds really cumbersome. And it is.

GitHub to the rescue — Part 2

Here’s where we use GitHub’s slightly less known, but really powerful tool: GitHub Actions.

GitHub Actions is GitHub’s super powerful builtin CI/CD tool. It runs a virtual machine for you, that can pull your code, test it, and deploy it, every single time you push an update to your repository. All for no cost at all.

What we are going to do is use its Python workflow that runs our HTML generator script for us, and use a plugin to create a separate GitHub Pages branch to deploy it to, automatically.

Using this, we can update our short links by simply editing and saving our links.json file, everything else will be handled for us by GitHub.

But first, we need to make some arrangements.

Deploy Keys: a short explanation

GitHub has a feature called Deploy Keys, which are essentially just SSH keys that have read/write access to your repository, so a computer that has that key can push code to your repo directly.

To add a deploy key to your GitHub repo, follow these simple steps:

Generate an SSH key: open your terminal/command prompt and type ssh-keygen Give it any name you like (using the repository name would be good), use an empty passphrase for it, and keep everything else default. This will generate a private key file, and a public .pub file, both with the same name. (If the command _ssh-keygen_ doesn’t work, you might have to install OpenSSH on your system)

You just generated an SSH public-private key pair!

Open your repository settings, and go to the Deploy Keys tab. It should look something like this:

Click on “Add deploy key” on the top right, and copy-paste the entire contents of the .pub file that you just generated. (Make sure to click the “write access” checkbox):

The title of the key doesn’t really matter here.

Then head down to the Secrets section of your repository settings, and add a new secret named ACTIONS_DEPLOY_KEY, and copy your private key into it. It should look something like:

That’s a long secret.

And that’s it! We can now get into the Action (pun intended).

Final step: Setting up the GitHub Action

The workflow we will be using

Simply navigate to the Actions tab, and click on the Python Application workflow. Replace its contents with the ones provided below:

And we’re done!

You can create your own free URL shortener by simply forking my repository:

https://github.com/tusharsadhwani/url-shortener,

adding a secret and a deploy key to your repo with write access, and changing the website name from tshr.me to whatever your domain name is, in the main.py file.

Thanks for reading this article!

Big thanks from my side to Coding Garden, whose videos motivated me to make this project in the first place, and this article, which led me to the free hack that I came up with. Also thanks to Wojtek Pawlik (GingerPlusPlus) for suggesting a non-javascript method for redirects.

Forem: Tushar Sadhwani

The math behind Python's slices

The basics

The interesting bits

Negative numbers in slices

Negative step

Summary

What the f-strings?

Index

What is an f-string?

Simple examples

A brief history of string formatting

Why f-strings?

More Examples

Padding / truncating

Alignment

Rounding numbers

Converting types

Adding commas to large numbers

Special syntax

!s !r and !a

datetime formatting

{x=} syntax

Nested formatting

Custom formatting using __format__

Limitations

Conclusion

Ace your LeetCode preparations

python-leetcode-runner

An example workflow

Advanced use cases

Validator solution example

Extra tips

Footnotes

I fixed Google's scripting tool

Introducing zxpy

The Comprehensive Guide to mypy

Read the original on my personal blog

Index

Setting up mypy

Using mypy in the terminal

Using mypy in VSCode

Primitive types

Collection types

Type debugging - Part 1

Union and Optional

Any type

Miscellaneous types

Tuple

TypedDict

Literal

Final

NoReturn

Typing classes

Typing namedtuples

Typing decorators

Typing generators

Typing *args and **kwargs

Duck types

Function overloading with @overload

Type type

Typing pre-existing projects

Type debugging - Part 2

Typing Context managers

Typing async functions

Generics

Generic functions

Generic classes

Generic types

Advanced/Recursive type checking with Protocol

Further learning

Does Python need types?

Why types?

Self-documenting code

None-awareness

Better autocompletion and IDE support

Where can I use it?

Conclusion

Connecting Android Apps to localhost, Simplified

The Bad way

Negative `step`

`datetime` formatting

`{x=}` syntax

Custom formatting using `format`

Typing `*args` and `**kwargs`

Function overloading with `@overload`

`Type` type

Advanced/Recursive type checking with `Protocol`

`None`-awareness