Forem: Miguel Brito

15 Easy Ways to Trim a String in Python

Miguel Brito — Sat, 29 Jan 2022 17:38:19 +0000

I'm not gonna lie. There are multiple ways you can trim a string in Python.

But... the truth is, you don't need to know every one of them.

In this article, you'll see only the most important techniques, such as stripping leading and trailing spaces (as well as the ones inside the string). You'll also learn how to remove tabs, newlines, carriage return (CRLF), and other characters. And we'll be using nothing more than native methods and regex—no external libraries required!

By the end of this article, you'll have mastered:

How to trim a string
- by stripping leading whitespace from the beginning
- by stripping trailing whitespace from the end
- by removing spaces the start and end of a string
How trim newlines
How trim carriage return (CRLF)
How trim tabs
How to trim a combination of characters from a string
How to remove multiple spaces inside a string
- by removing only duplicates
- by removing all spaces
How to strip a list of strings
How to strip an (Numpy) array of strings

How to Trim Characters From a String

Trimming a string means deleting certain chars from the start, the end, or both sides of a string. Removing unwanted chars makes it easier to compare strings and can prevent hard to debug issues.

You can remove any kind o character, but usually what we're interested in is deleting blank spaces, new lines, carriage return (CRLF), tabs and other special symbols.

In this section, we're going to see how to remove leading or trailing spaces, blank spaces, newline character, carriage return (CRLF), and tabs.

Stripping Leading Whitespace From Beginning of a String

The str class has a very convenient method to trim leading spaces named str.lstrip, a shorthand for "left-strip", since it trims a string from the left-hand side. You can think of it as a left trim.

>>> '   hello   '.lstrip()
'hello   '

When calling str.lstrip with no arguments, it removes all whitespaces from left to right. But if all you want is to strip the first char, then there are two ways of doing this. The first one assumes that there will always be at least one whitespace in the beginning of the string. If that's the case, then you can just slice it.

>>> s = '  hello'
>>> s = s[1:]
>>> s
' hello'

If there's no guarantee of that, we'll need to check first if the string starts with space.

>>> def strip_first(s: str, ch: str = ' ') -> str:
     if s and s[0] == ch:
         return s[1:]
     return s

>>> strip_first('hello')
'hello'

>>> strip_first('   hello')
 '  hello'

Stripping Trailing Whitespace From End of a String

The way to remove trailing spaces from the end of the string is to use str.rstrip.

This method expects a list of chars and trims the string from the right. It removes all chars that match one of those you passed, and stop as soon as it cannot match anymore. By default, str.rstrip() removes blanks if you don't pass anything to it. You can think of it as a right trim.

>>> '   hello   '.rstrip()
'   hello'
>>> '***hello***'.rstrip('*')
'***hello'

Sometimes you might want to trim only the last character of a string. And we can use the same logic from the previous example. Check if the last char is a space, and use slice to remove it.

>>> def strip_last(s: str, ch: str = ' ') -> str:
     if s and s[-1] == ch:
         return s[:-1]
     return s


>>> strip_last('hello')
'hello'

>>> strip_last('hello ')
'hello'

>>> strip_last('')
''

Removing Spaces From From Start and End of a String

If all you want is to remove whitespaces from start and end of string, str.strip will serve you better.

This method trims both sides of the string. And just like str.lstrip and str.rstrip, if you can pass any combination of chars as argument, it removes them from both ends.

⚠️ A common misconception is to think that there's a trim() function in Python.

# by default, strip removes whitespaces
>>> '   hello   '.strip()
'hello'
# but you can also strip other character
>>> '***hello***'.strip('*')
'hello'

How to Trim Newlines

We've seen how str.strip can remove blank spaces from both sides of a string. I've also mentioned that this method takes a chars argument that you can use pass a combination of character you want to trim.

To trim line breaks, you can pass \n and it will strip all newlines from both sides of the string.

>>> s = """
... 
... 
...  hello
... 
... 
... """
>>> s
'\n\n\n hello\n\n\n'
>>> s.strip('\n')
' hello'

How to Trim Carriage Return (CRLF)

The Carriage Return (CR), and Line Feed (LF) are nothing more than a newline character. They are represented by the concatenation of \r and \n forming \r\n. This is how Microsoft Windows, Symbian OS and other non-Unix operating systems represent a new line [source].

Removing them from a string is the same as removing the single newline. You feed str.strip with \r\n and method does its job!

>>> s = "  hello world\r\n\r\n"
>>> print(s)
  hello world


>>> s.strip('\r\n')
'  hello world'

How to Trim Tabs

If you are following this guide from the beginning you might already know how to do this. Trimming tabs from a string in Python is the same as other characters, you use str.strip and pass the '\t' string to it.

>>> s = "\t\t\t  hello  world \t"       
>>> s
'\t\t\t  hello  world \t'
>>> print(s)
              hello  world  
>>> s.strip('\t')
'  hello  world '

And that's it!

How to Trim a Combination of Characters From a String

As I mentioned before, str.strip takes as argument a string, not just a single char. This sequence of chars is a combination of all chars you want to remove from the beginning and end of your string.

>>> s = "  \ns hello world \n    s"
>>> s    
'  \ns hello world \n    s'
>>> print(s)

s hello world 
    s
>>> s.strip('\n s')
'hello world'

How to Remove Multiple Spaces Inside a String

Sometimes you want to do more than trimming, let's say you want to remove chars inside the string. There are two ways of doing this: one is to remove only the duplicates; the other is to remove all extra spaces.

Removing Only Duplicates

To remove only the duplicated characters, you can use the regex module re

>>> import re
>>> s = "   Python   is really   a    great language.    "
>>> re.sub("\s+" , " ", s)
' Python is really a great language. '

This method gets rid of all consecutive spaces. What if you want to do not only that, but also trim the string by removing the leading and trailing blanks?

One way is to split the string and then joining then like so:

>>> s = "   Python   is really   a    great language.    "
>>> " ".join(s.split())
'Python is really a great language.'
>>> # This is the same as using regex then stripping the whitespaces
>>> re.sub("\s+" , " ", s).strip()
'Python is really a great language.'

Removing All Spaces

Now, if you want to strip all whitespace in your string, either use regex or call the str.replace method.

Using `re` (regex module)

>>> import re
>>> s = "   Python   is really   a    great language.    "
>>> re.sub("\s+" , "", s) 
'Pythonisreallyagreatlanguage.'

Using `replace`

>>> s = "   Python   is really   a    great language.    "
>>> s.replace(' ', '')
'Pythonisreallyagreatlanguage.'

How to Strip a List of Strings

Trimming a list of strings is almost the same as trimming an individual one. The only difference is that you have to iterate over the list, and call str.strip method on each one. You do so by using a list comprehension, for example, to return a new list with all strings trimmed.

>>> lst = ["string1\n", "string2\n", "string3\n"]
>>> [s.strip('\n') for s in lst]
['string1', 'string2', 'string3']

How to Strip an (Numpy) Array of Strings

It's very common to use Numpy for data science tasks due to its performance and ease to use.

If you have a array of strings and want to trim each one of them, Numpy comes with an efficient vectorized implementation of strip.

In fact, it also has .lstrip, .rstrip, .replace, and many other string operations.

The vectorized versions work slightly differently, they are not a method but a function in the numpy.char module. So you must pass the array and the list of chars you want to trim.

>>> import numpy as np
>>> arr = np.array([' helloworld   ', ' hello'])
array([' helloworld   ', ' hello'], dtype='<U7')
>>> np.char.strip(arr, ' ')
array(['helloworld', 'hello'], dtype='<U7')

Conclusion

In this post, you learned several ways of trimming a string in Python, including array of strings. Python allows us to strip leading and trailing characters easily. And if instead of removing the extra chars on each side you want to remove the ones internally, you can count on the regex module. I hope you've found this article helpful and see you next time!

References:

https://stackoverflow.com/questions/761804/how-do-i-trim-whitespace-from-a-string

https://stackoverflow.com/questions/8270092/remove-all-whitespace-in-a-string

https://stackoverflow.com/questions/1546226/is-there-a-simple-way-to-remove-multiple-spaces-in-a-string

Other posts you may like:

See you next time!

This post was originally published at https://miguendes.me

How to Use datetime.timedelta in Python With Examples

Miguel Brito — Wed, 15 Dec 2021 08:50:04 +0000

In this tutorial, you'll learn how to use datetime.timedelta to perform date arithmetic.

With timedelta you can add days, minutes, seconds, hours, weeks and many more to a datetime.date or a datetime.datetime object.

You'll also learn how to:

convert a timedelta to seconds, minutes, hours, or days
convert a time delta to years
how to take the difference between two dates
how to format a time delta as string

Let's go!

Adding Seconds, Minutes, Hours, Days, Weeks and Whatnot to a Date
- How to Use timedelta to Add Days to a date or datetime Object
- How to Use timedelta to Add Minutes to a datetime Object
- How to Use timedelta to Add Weeks, Hours, Seconds, Milliseconds to a datetime
- How to Add Years to a datetime in Python
- How to Add Months to a datetime in Python
How to convert a timedelta to seconds, minutes, hours, or days
How to Take the Difference Between Two Dates
- How to Calculate the Number of Days Between Two Dates
- How to Calculate the Number of Minutes Between Two Dates
How to Format a timedelta as string
Conclusion

Adding Seconds, Minutes, Hours, Days, Weeks and Whatnot to a Date

A timedelta object denotes a duration, it can also represent the difference between two dates or times.

We can use this object to add or subtract a duration from a date and it defines its constructor as datetime.timedelta(days=0, seconds=0, microseconds=0, milliseconds=0, minutes=0, hours=0, weeks=0). As you can see, all arguments are optional and default to 0. It can take ints or floats, positive or negative.

Even though you can pass weeks, hours, minutes and milliseconds only days, seconds, and microseconds are stored internally.

In this section, we'll see basic arithmetic operations such as adding/subtracting a duration to/from a date.

How to Use `timedelta` to Add Days to a `date` or `datetime` Object

Since timedelta represents a duration, we can use it to add days to a datetime. The number of can be positive or negative, thus allowing us to create a date in the future or in the past. The code snippet below shows an example.

>>> import datetime

>>> now = datetime.datetime.now()

>>> now
datetime.datetime(2020, 11, 3, 22, 5, 21, 979147)

>>> from datetime import timedelta

>>> now + timedelta(days=3)
datetime.datetime(2020, 11, 6, 22, 5, 21, 979147)

>>> now + timedelta(days=-3)
datetime.datetime(2020, 10, 31, 22, 5, 21, 979147)

As you can see, adding a positive number of days yields a future date whereas adding a negative number brings the date to the past.

If you want to add days to a date object, the process is the same.

>>> today = datetime.date.today()

>>> today
datetime.date(2020, 11, 5)

>>> today + timedelta(days=3)
datetime.date(2020, 11, 8)

How to Use `timedelta` to Add Minutes to a `datetime` Object

Since timedelta object sets all arguments to 0 by default, we have the option to set only the ones we need. This allows us to add only minutes, for instance.

>>> now
datetime.datetime(2020, 11, 3, 22, 5, 21, 979147)

>>> now + timedelta(minutes=3)
datetime.datetime(2020, 11, 3, 22, 8, 21, 979147)

>>> now + timedelta(minutes=-3)
datetime.datetime(2020, 11, 3, 22, 2, 21, 979147)

How to Use `timedelta` to Add Weeks, Hours, Seconds, Milliseconds to a `datetime`

Adding weeks, seconds, milliseconds and even microseconds works in a similar fashion.

>>> now + timedelta(weeks=3)
datetime.datetime(2020, 11, 24, 22, 5, 21, 979147)

>>> now + timedelta(hours=3)
datetime.datetime(2020, 11, 4, 1, 5, 21, 979147)

>>> now + timedelta(microseconds=3)
datetime.datetime(2020, 11, 3, 22, 5, 21, 979150)

>>> now + timedelta(milliseconds=3)
datetime.datetime(2020, 11, 3, 22, 5, 21, 982147)

How to Add Years to a `datetime` in Python

It's definitely possible to use timedelta to add years to a datetime, but some things can go wrong and it's easy to shoot yourself in the foot. For example, you need to take into account leap years yourself. IMHO, the best way to add a certain number of years to a datetime is by using the dateutil library.

>>> import datetime

>>> from dateutil.relativedelta import relativedelta

>>> now = datetime.datetime.now()

>>> now
datetime.datetime(2020, 11, 4, 22, 9, 5, 672091)

>>> now + relativedelta(years=2)
datetime.datetime(2022, 11, 4, 22, 9, 5, 672091)

How to Add Months to a `datetime` in Python

The same problem arises when we need to add months to a datetime using timedelta. Adding months are not supported by default and requires manual calculation. You can use days, but you’d need to know how many days that month has and so. In a nutshell, it’s too error prone. Again, the best you can do is to use dateutil.relativedelta.

>>> from dateutil.relativedelta import relativedelta
>>> now
datetime.datetime(2020, 11, 4, 22, 9, 5, 672091)

>>> now + relativedelta(years=2)
datetime.datetime(2022, 11, 4, 22, 9, 5, 672091)

>>> now + relativedelta(months=12)
datetime.datetime(2021, 11, 4, 22, 9, 5, 672091)

>>> now + relativedelta(months=24)
datetime.datetime(2022, 11, 4, 22, 9, 5, 672091)

How to convert a `timedelta` to seconds, minutes, hours, or days

A timedeltaobject allows adding a delta to a datetime but sometimes is useful to convert it into a single time unit, such as seconds, or minutes.

In this section, we'll explore how to do that.

How to convert a `timedelta` to seconds

A timedelta has only one method called timedelta.total_seconds(). This method returns the total number of seconds the duration has. If we want to convert a timedelta object to seconds, we can just call it.

>>> import datetime

>>> delta = datetime.timedelta(days=1, seconds=34)

# a day has 24h, each hour has 60min of 60s = 24*60*60 = 86400
# 86400s + 34s = 86434s
>>> delta.total_seconds()
86434.0

What is the difference between total_seconds() and timedelta.seconds?

total_seconds()—as its name implies—corresponds to total seconds within the whole duration. On the flip side, timedelta.seconds is an internal property that represents the number of seconds within a day.

To be more precise, timedelta.seconds stores the seconds if it is less than a day, that is, from 0 to 86399. Otherwise, if the number of seconds is greater than 86399, timedelta converts this number to days, or weeks as you'll see in the next example.

>>> import datetime

>>> delta = datetime.timedelta(seconds=34)

# Delta is withtin 0 and 86399, so delta.seconds returns that number
>>> delta.seconds
34

>>> delta = datetime.timedelta(days=1, seconds=34)

# 1 days + 34s = 86434s, so it overflows to days
>>> delta.seconds
0
>>> delta.days
1
# total_seconds returns 1 day in seconds + 34s = 86434s
>>> delta.total_seconds()
86434.0

How to convert a `timedelta` to minutes

To convert a timedelta to minutes you need to use a bit of math. Unfortunately, timedelta does not provide any way of accessing the number of minutes in a duration. In the end, you need to do the conversion yourself.

There are two different ways of doing this conversion:

the first one you divide the total_seconds() by the number of seconds in a minute, which is 60
the second approach, you divide the timedelta object by timedelta(minutes=1)

>>> import datetime

>>> delta = datetime.timedelta(hours=3, minutes=13, seconds=34)

# there's NO minutes in a time delta object
>>> delta.minutes
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-31-b45e912051b9> in <module>
----> 1 delta.minutes

AttributeError: 'datetime.timedelta' object has no attribute 'minutes'

# we set a variable to represent the number of seconds in a minute
>>> NUM_SECONDS_IN_A_MIN = 60

# we then divide the total seconds by the number of seconds in a minute
# this gives us around 193 minutes
>>> delta.total_seconds() / NUM_SECONDS_IN_A_MIN
193.56666666666666

# alternatively, we use the divide operator
>>> delta / datetime.timedelta(minutes=1)
193.56666666666666

How to convert a `timedelta` to hours

We can follow the same logic to convert a timedelta to hours. Instead of dividing the total_seconds() by the number of seconds in a minute, or dividing thetimedelta object by timedelta(minutes=1), we do it for hour.

>>> import datetime

>>> delta = datetime.timedelta(hours=3, minutes=13, seconds=34)

delta.hours
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-3-8c2202cab691> in <module>
----> 1 delta.hours

AttributeError: 'datetime.timedelta' object has no attribute 'hours'

>>> delta / datetime.timedelta(hours=1)
3.226111111111111

How to convert a `timedelta` to days

Converting a timedelta to days is easier, and less confusing, than seconds. According to the docs, only days, seconds and microseconds are stored internally. To get the number of days in a time delta, just use the timedelta.days.

⚠️ WARNING: timedelta.days is an internal property that is not listed in the docs, so it's not a good idea to rely on it. A more robust approach is to divide the time delta object by datetime.timedelta(days=1).

>>> import datetime

>>> delta = datetime.timedelta(weeks=2, days=3, seconds=34)

# 1 week has 7 days, so 2 weeks has 14 days. 2 weeks + 3 days = 17 days
>>> delta.days
17

# if you want the days including the fraction of seconds, divide it by timedelta(days=1)
>>> delta / datetime.timedelta(days=1)
17.000393518518518

How to convert a `timedelta` to years

If you've been following this guide since the beginning you might have started to pick up a pattern. However, I have bad news.

In the "what is timedelta?" section, I mentioned that you can create a timedelta by passing a combination of days, seconds, microseconds, milliseconds, minutes, hours, and weeks.

By default, timedelta doesn't support years. To do that we would need to calculate how many weeks there is in how many years we want to pass. It's definitely possible to use timedelta to add years to a datetime, but some things can go wrong. For example, you need to take into account leap years yourself.

The idea here is to create a variable that holds the number of seconds in a year. A full year has 365 days, but to account for the leap years, we add 0.25 to it, so 365.25. Each day has 24 hours of 60 min, and each minute has 60s. Multiply everything and you get the number of seconds in a year.

>>> import datetime

# 1 year has 52 weeks, so we create a delta of 2 years with 2*52
>>> delta = datetime.timedelta(weeks=2*52, days=3, seconds=34)

>>> delta
datetime.timedelta(days=731, seconds=34)

>>> def timedelta_to_years(delta: datetime.timedelta) -> float:
        seconds_in_year = 365.25*24*60*60
        return delta.total_seconds() / seconds_in_year

>>> timedelta_to_years(delta)
2.0013700027885517

# round to int, if you don't care about the fraction
>>> int(timedelta_to_years(delta))
2

Another alternative—to me the best one—is to get a delta duration in years is by using the python-dateutil library.

>>> import datetime

>>> from dateutil.relativedelta import relativedelta

>>> delta = relativedelta(years=2, weeks=3, months=1)

>>> delta
relativedelta(years=+2, months=+1, days=+21)

>>> delta.years
2

>>> delta = relativedelta(years=2, weeks=3, months=15)

>>> delta.years
3

How to Take the Difference Between Two Dates

As discussed earlier, timedelta can also represent the difference between two dates. The following sub-sections illustrate how you can do that.

How to Calculate the Number of Days Between Two Dates

To obtain the difference between two datetime objects in days you can use the - operator, for example.

>>> now = datetime.datetime.now()

>>> now
datetime.datetime(2020, 11, 3, 22, 36, 21, 674967)

>>> yesterday = datetime.datetime(2020, 11, 2)

>>> now - yesterday
datetime.timedelta(days=1, seconds=81381, microseconds=674967)

>>> now - yesterday
 datetime.timedelta(days=1, seconds=81381, microseconds=674967)

>>> (now - yesterday).days
1

How to Calculate the Number of Minutes Between Two Dates

This one requires more work, and we can achieve it in two different ways. The first one is using divmod and the second one is using timedelta.

According to the docs , divmod takes two (non complex) numbers as arguments and return a pair of numbers consisting of their quotient and remainder when using integer division. In our case, we want to divide the total number of seconds contained in the duration by the number of seconds in one minute, which is 60.

>>> now = datetime.datetime.now()

>>> now
datetime.datetime(2020, 11, 3, 22, 57, 12, 300437)

>>> yesterday = datetime.datetime(2020, 11, 2, 22, 57, 12, 300437)

>>> diff = now - yesterday

>>> diff.total_seconds()
86400.0

>>> diff / timedelta(minutes=1)
1440.0

>>> divmod(diff.total_seconds(), 60)
(1440.0, 0.0)

>>> int(diff / timedelta(minutes=1))
1440

How to Format a `timedelta` as string

Sometimes we want to get a string representation of a timedelta object. Even though you can do that by calling str(timedelta_obj), sometimes the result will not be good. The reason is that it can vary depending on the length of the duration the object represents. For example, take a look at what happens when you try to print different timedeltas.

>>> from datetime import timedelta

>>> timedelta(seconds=123)
datetime.timedelta(seconds=123)

>>> str(timedelta(seconds=123))
'0:02:03'

>>> str(timedelta(seconds=123456))
'1 day, 10:17:36'

>>> str(timedelta(seconds=1234.56))
'0:20:34.560000'

With that in mind, the question is: how can we have a more consistent format?

Sadly, we don’t have many options other than implementing a formatting function ourselves. The good thing is, that’s not so hard.

Suppose we want to print the timedelta in this format: [N days] %H:%M:%S. One way to do that is using python’s f-strings.

def format_timedelta(delta: timedelta) -> str:
    """Formats a timedelta duration to [N days] %H:%M:%S format"""
    seconds = int(delta.total_seconds())

    secs_in_a_day = 86400
    secs_in_a_hour = 3600
    secs_in_a_min = 60

    days, seconds = divmod(seconds, secs_in_a_day)
    hours, seconds = divmod(seconds, secs_in_a_hour)
    minutes, seconds = divmod(seconds, secs_in_a_min)

    time_fmt = f"{hours:02d}:{minutes:02d}:{seconds:02d}"

    if days > 0:
        suffix = "s" if days > 1 else ""
        return f"{days} day{suffix} {time_fmt}"

    return time_fmt

>>> format_timedelta(timedelta(hours=23, seconds=3809))
'1 day 00:03:29'

>>> format_timedelta(timedelta(hours=23))
'23:00:00'

>>> format_timedelta(timedelta(hours=25))
'1 day 01:00:00'

>>> format_timedelta(timedelta(hours=48, seconds=3700))
'2 days 01:01:40'

Conclusion

That’s it for today, folks! I hope you’ve learned something different and useful. Knowing how to perform date calculations such as addition and subtraction is very important. The timedelta object is good enough for most situations but if you need more complex operations go for the dateutil library.

Other posts you may like:

See you next time!

This post was originally published at https://miguendes.me

7 Different Ways to Flatten a List of Lists in Python

Miguel Brito — Sat, 25 Sep 2021 11:10:42 +0000

Ever wondered how can you flatten, or unnest, a 2D list of lists in Python?

In another words, how to turn 2-D lists into 1D:

[[1, 2], [3, 4], [5, 6, 7], [8]] -> [1, 2, 3, 4, 5, 6, 7, 8]?
[[1, 2], [4, 5], [[[7]]], [[[[8]]]]] -> [1, 2, 3, 4, 5, 6, 7, 8]?
[1, 2, 3, [4, 5, 6], [[7, 8]]] -> [1, 2, 3, 4, 5, 6, 7, 8]?

What about lists with mixed types such as list of strings, or list of tuples?

[[1, 2], "three", ["four", "five"]] -> [1, 2, "three", "four", "five"]
[[1, 2], (3, 4), (5, 6, 7), [8]] -> [1, 2, 3, 4, 5, 6, 7, 8]

In this post, we’ll see how we can unnest a arbitrarily nested list of lists in 7 different ways. Each method has pros and cons, and varied performance. You’ll learn how to identify the most appropriate solution for your problem by creating your own flatten() function in Python.

For all examples, we'll use Python 3, and for the tests pytest.

By the end of this guide, you'll have learned:

how to flatten / unnest a list of mixed types, including list of strings, list of tuples or ints
the best way to flatten lists of lists with list comprehensions
how to unfold a list and remove duplicates
how to convert a nested list of lists using the built-in function sum from the standard library
how to use numpy to flatten nested lists
how to use itertools chain to create a flat list
the best way to flatten a nested list using recursion or without recursion

Flattening a list of lists with list comprehensions
How to flatten list of strings, tuples or mixed types
How to flatten a nested list and remove duplicates
Flattening a nested list of lists with the sum function
Flattening using itertools.chain
Flatten a regular list of lists with numpy
Flattening irregular lists
The recursive approach
The iterative approach
Which method is faster? A performance comparison
Conclusion

Flattening a list of lists with list comprehensions

Let’s imagine that we have a simple list of lists like this [[1, 3], [2, 5], [1]] and we want to flatten it.

In other words, we want to convert the original list into a flat list like this [1, 3, 2, 5, 1]. The first way of doing that is through list/generator comprehensions. We iterate through each sublist, then iterate over each one of them producing a single element each time.

The following function accepts any multidimensional lists as an argument and returns a generator. The reason for that is to avoid building a whole list in memory. We can then use the generators to create a single list.

To make sure everything works as expected, we can assert the behavior with the test_flatten unit test.

from typing import List, Any, Iterable


def flatten_gen_comp(lst: List[Any]) -> Iterable[Any]:
    """Flatten a list using generators comprehensions."""
    return (item
            for sublist in lst
            for item in sublist)


def test_flatten():
    lst = [[1, 3], [2, 5], [1]]

    assert list(flatten_gen_comp(lst)) == [1, 3, 2, 5, 1]

This function returns a generator, to get a list back we need to convert the generator to list.

When we run the test we can see it passing...

============================= test session starts ==============================

flatten.py::test_flatten PASSED                                          [100%]

============================== 1 passed in 0.01s ===============================

Process finished with exit code 0

If you prefer you can make the code shorter by using a lambda function.

>>> flatten_lambda = lambda lst: (item for sublist in lst for item in sublist)

>>> lst = [[1, 3], [2, 5], [1]]

>>> list(flatten_lambda(lst))
[1, 3, 2, 5, 1]

How to flatten list of strings, tuples or mixed types

The technique we've seen assumes the items are not iterables. Otherwise, it flattens them as well, which is the case for strings. Let's see what happens if we plug a list of lists and strings.

from typing import List, Any, Iterable


def flatten_gen_comp(lst: List[Any]) -> Iterable[Any]:
    """Flatten a list using generators comprehensions."""
    return (item
            for sublist in lst
            for item in sublist)


>>> lst = [['hello', 'world'], ['my', 'dear'], 'friend']
>>> list(flatten(lst))
['hello', 'world', 'my', 'dear', 'f', 'r', 'i', 'e', 'n', 'd']

Oops, that's not what we want! The reason is, since one of the items is iterable, the function will unfold them as well.

One way of preventing that is by checking if the item is a list of not. If not, we don't iterate over it.

>>> from typing import List, Any, Iterable


>>> def flatten(lst: List[Any]) -> Iterable[Any]:
    """Flatten a list using generators comprehensions.
        Returns a flattened version of list lst.
    """

    for sublist in lst:
         if isinstance(sublist, list):
             for item in sublist:
                 yield item
         else:
             yield sublist

>>> lst = [['hello', 'world'], ['my', 'dear'], 'friend']

>>> list(flatten(l))
['hello', 'world', 'my', 'dear', 'f', 'r', 'i', 'e', 'n', 'd']

Since we check if sublist is a list of not, this works with list of tuples as well, any list of iterables, for that matter.

>>> lst = [[1, 2], 3, (4, 5)]
>>> list(flatten(lst))
[1, 2, 3, (4, 5)]

Lastly, this flatten function works for multidimensional list of mixed types.

>>> lst = [[1, 2], 3, (4, 5), ["string"], "hello"]

>>> list(flatten(lst))
[1, 2, 3, (4, 5), 'string', 'hello']

How to flatten a list and remove duplicates

To flatten a list of lists and return a list without duplicates, the best way is to convert the final output to a set.

The only downside is that if the list is big, there'll be a performance penalty since we need to create the set using the generator, then convert set to list.

>>> from typing import List, Any, Iterable


>>> def flatten(lst: List[Any]) -> Iterable[Any]:
    """Flatten a list using generators comprehensions.
        Returns a flattened version of list lst.
    """

    for sublist in lst:
         if isinstance(sublist, list):
             for item in sublist:
                 yield item
         else:
             yield sublist

>>> lst = [[1, 2], 3, (4, 5), ["string"], "hello", 3, 4, "hello"]

>>> list(set(flatten(lst)))
[1, 2, 3, 'hello', 4, (4, 5), 'string']

Flattening a list of lists with the `sum` function

The second strategy is a bit unconventional and, truth to be told, very "magical".

Did you know that we can use the built-in function sum to create flattened lists?

All we need to do is to pass the list as an argument along an empty list. The following code snippet illustrates that.

If you’re curious about this approach, I discuss it in more detail in another post.

from typing import List, Any, Iterable


def flatten_sum(lst: List[Any]) -> Iterable[Any]:
    """Flatten a list using sum."""
    return sum(lst, [])


def test_flatten():
    lst = [[1, 3], [2, 5], [1]]

    assert list(flatten_sum(lst)) == [1, 3, 2, 5, 1]

And the tests pass too...

flatten.py::test_flatten PASSED

Even though it seems clever, it's not a good idea to use it in production IMHO. As we'll see later, this is the worst way to flatten a list in terms of performance.

Flattening using `itertools.chain`

The third alternative is to use the chain function from the itertools module. In a nutshell, chain creates a single iterator from a sequence of other iterables. This function is a perfect match for our use case.

Equivalent implementation of chain taken from the official docs looks like this.

import itertools


def chain(*iterables):
    # chain('ABC', 'DEF') --> A B C D E F
    for it in iterables:
        for element in it:
            yield element

We can then flatten the multi-level lists like so:

def flatten_chain(lst: List[Any]) -> Iterable[Any]:
    """Flatten a list using chain."""
    return itertools.chain(*lst)


def test_flatten():
    lst = [[1, 3], [2, 5], [1]]

    assert list(flatten_chain(lst)) == [1, 3, 2, 5, 1]

And the test pass...

[OMITTED]
....
flatten.py::test_flatten PASSED   
...
[OMITTED]

Flatten a regular list of lists with numpy

Another option to create flat lists from nested ones is to use numpy. This library is mostly used to represent and perform operations on multidimensional arrays such as 2D and 3D arrays.

What most people don't know is that some of its function also work with multidimensional lists or other list of iterables. For example, we can use the numpy.concatenate function to flatten a regular list of lists.

>>> import numpy as np

>>> lst = [[1, 3], [2, 5], [1], [7, 8]]

>>> list(np.concatenate(lst))
[1, 3, 2, 5, 1, 7, 8]

Flattening irregular lists

So far we've been flattening regular lists, but what happens if we have a list like this [[1, 3], [2, 5], 1] or this [1, [2, 3], [[4]], [], [[[[[[[[[5]]]]]]]]]]?

Unfortunately, that ends up not so well if we try to apply any of those preceding approaches. In this section, we’ll see two unique solutions for that, one recursive and the other iterative.

The recursive approach

Solving the flatting problem recursively means iterating over each list element and deciding if the item is already flattened or not. If so, we return it, otherwise we can call flatten_recursive on it. But better than words is code, so let’s see some code.

from typing import List, Any, Iterable


def flatten_recursive(lst: List[Any]) -> Iterable[Any]:
    """Flatten a list using recursion."""
    for item in lst:
        if isinstance(item, list):
            yield from flatten_recursive(item)
        else:
            yield item

def test_flatten_recursive():
    lst = [[1, 3], [2, 5], 1]

    assert list(flatten_recursive(lst)) == [1, 3, 2, 5, 1]

Bare in mind that the if isinstance(item, list) means it only works with lists. On the other hand, it will flatten lists of mixed types with no trouble.

>>> list(flatten_recursive(lst))
[1, 2, 3, (4, 5), 'string', 'hello']

The iterative approach

The iterative approach is no doubt the most complex of all of them.

In this approach, we’ll use a deque to flatten the irregular list. Quoting the official docs:

"Deques are a generalization of stacks and queues (the name is pronounced “deck” and is short for “double-ended queue”). Deques support thread-safe, memory efficient appends and pops from either side of the deque with approximately the same O(1) performance in either direction."

In other words, we can use deque to simulate the stacking operation of the recursive solution.

To do that, we’ll start just like we did in the recursion case, we’ll iterate through each element and if the element is not a list, we’ll append to the left of the deque. The appendleft method append the element to the leftmost position of the deque, for example:

>>> from collections import deque

>>> l = deque()

>>> l.appendleft(2)

>>> l
deque([2])

>>> l.appendleft(7)

>>> l
deque([7, 2])

If the element is a list, though, then we need to reverse it first to pass it to “extendleft” method, like this my_deque.extendleft(reversed(item)). Again, similar to a list, extendleft adds each item to the leftmost position of the deque in series. As a result, deque will add the elements in a reverse order. That’s exactly why we need to reverse the sub-list before extending left. To make things clearer, let’s see an example.

>>> l = deque()

>>> l.extendleft([1, 2, 3])

>>> l
deque([3, 2, 1])

The final step is to iterate over the deque removing the leftmost element and yielding it if it’s not a list. If the element is a list, then we need to extend it left. The full implement goes like this:

from typing import List, Any, Iterable


def flatten_deque(lst: List[Any]) -> Iterable[Any]:
    """Flatten a list using a deque."""
    q = deque()
    for item in lst:
        if isinstance(item, list):
            q.extendleft(reversed(item))
        else:
            q.appendleft(item)
        while q:
            elem = q.popleft()
            if isinstance(elem, list):
                q.extendleft(reversed(elem))
            else:
                yield elem

def test_flatten_super_irregular():
    lst = [1, [2, 3], [4], [], [[[[[[[[[5]]]]]]]]]]

    assert list(flatten_deque(lst)) == [1, 2, 3, 4, 5]

When we run the test, it happily pass...

============================= test session starts ==============================
...

flatten.py::test_flatten_super_irregular PASSED                          [100%]

============================== 1 passed in 0.01s ===============================

This approach can also flatten lists of mixed types with no trouble.

>>> list(flatten_deque(lst))
[1, 2, 3, (4, 5), 'string', 'hello']

Which method is faster? A performance comparison

As we’ve seen in the previous section, if our multi-level list is irregular we have little choice. But assuming that this is not a frequent use case, how these approaches compare in terms of performance?

In this last part, we’ll run a benchmark and compare which solutions perform best.

To do that, we can use the timeit module, we can invoke it in IPython by doing %timeit [code_to_measure]. The following list is the timings for each one of them. As we can see, flatten_chain is the fastest implementation of all. It flatted our list lst in 267 µs avg. The slowest implementation is flatten_sum, taking around 42 ms to flatten the same list.

PS: A special thanks to @hynekcer who pointed out a bug in this benchmark. Since most of the functions return a generator, we need to consume all elements in order to get a better assessment. We can either iterate over the generator or create a list out of it.

Flatten generator comprehension

In [1]: lst = [[1, 2, 3], [4, 5, 6], [7], [8, 9]] * 1_000

In [2]: %timeit list(flatten_gen_comp(lst))
615 µs ± 2.74 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Flatten sum

In [3]: In [19]: %timeit list(flatten_sum(lst))
42 ms ± 660 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

Flatten chain

In [4]: %timeit list(flatten_chain(lst))
267 µs ± 517 ns per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Flatten numpy

In [5]: %timeit list(flatten_numpy(lst))
4.65 ms ± 14.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Flatten recursive

In [6]: %timeit list(flatten_recursive(lst))
3.02 ms ± 174 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Flatten deque

In [7]: %timeit list(flatten_deque(lst))
2.97 ms ± 21.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Conclusion

We can flatten a multi-level list in several different ways with Python. Each approach has its pros and cons. In this post we looked at 5 different ways to create 1D lists from nested 2D lists, including:

regular nested lists
irregular lists
list of mixed types
list of strings
list of tuples or ints
recursive and iterative approach
using the iterools module
removing duplicates

Other posts you may like:

Everything You Need to Know About Python's Namedtuples
The Best Way to Compare Two Dictionaries in Python
Python's f-strings: 73 Examples to Help You Master It
Design Patterns That Make Sense in Python: Simple Factory
How to Pass Multiple Arguments to a map Function in Python
3 Ways to Unit Test REST APIs in Python

See you next time!

This post was originally published at https://miguendes.me/python-flatten-list

How to Check if an Exception Is Raised (or Not) With pytest

Miguel Brito — Sat, 18 Sep 2021 09:05:58 +0000

TL;DR

Time is a precious resource so I won't waste yours. Here's how you can assert an exception is raised and how to check that in pytest.

Solution: Use pytest.raises

import pytest

def test_raises_exception():
    with pytest.raises(ZeroDivisionError):
        1 / 0

And here's how you assert no exception is raised.

Solution: Enclose your code in a try/except block and and if the code raises, you can catch it and print a nice message. pytest is smart enough to make the test fail even if you don't catch it but having a message makes your test cleaner.

def my_division_function(a, b):
    return a / b

def test_code_raises_no_exception():
    """
    Assert your python code raises no exception.    
    """
    try:
        my_division_function(10, 5)
    except ZeroDivisionError as exc:
        assert False, f"'10 / 5' raised an exception {exc}"

And that's it, if you want to know more, please follow along.

Introduction

In this tutorial, you'll learn how to use pytest to:

assert that an exception is raised
assert the exception message
assert the exception type
assert that an exception is not raised

In a nutshell, we'll see how to use pytest.raises for each of those cases with examples.

How to Assert That an Exception Is Raised
How to Assert That NO Exception Is Raised
How to Assert the Exception Message - And Type
Conclusion

How to Assert That an Exception Is Raised

In this section, I’m going to show you how you can assert that your code raises an exception. This is a frequent use case and can sometimes tricky. The wonderful thing is, if you are using pytest you can do that in an idiomatic and cleaner way.

Let’s imagine that we have a function that checks for some keys in a dictionary. If a key is not present, it should raise a KeyError. As you can see, this is very generic and doesn’t tell the users much about the error. We can make it cleaner by raising custom exceptions, with different messages depending on the field.

import pytest


class MissingCoordException(Exception):
    """Exception raised when X or Y is not present in the data."""


class MissingBothCoordException(Exception):
    """Exception raised when both X and Y are not present in the data."""


def sum_x_y(data: dict) -> str:
    return data["x"] + data["y"]

Now, time to test this. How can we do that with pytest?

This code is deliberately wrong, as you can see we’re not raising anything. In fact, we want to see test failing first, almost like TDD. After seeing the test failing, we can fix our implementation and re-run the test.

def test_sum_x_y_missing_both():
    data = {"irrelevant": 1}
    with pytest.raises(MissingBothCoordException):
        sum_x_y(data)

Then we get the following output:

============================ FAILURES ============================
________________ test_sum_x_y_missing_both _________________

    def test_sum_x_y_missing_both():
        data = {"irrelevant": 1}
        with pytest.raises(MissingBothCoordException):
>           sum_x_y(data)

test_example.py:33: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

data = {'irrelevant': 1}

    def sum_x_y(data: dict) -> str:
>       return data["x"] + data["y"]
E       KeyError: 'x'

test_example.py:27: KeyError
==================== short test summary info =====================
FAILED test_example.py::test_sum_x_y_missing_both - KeyEr...
======================= 1 failed in 0.02s ========================

Ok, this makes sense, now it’s time to fix it. We’ll check if the data dict has both x and y, otherwise we raise a MissingBothCoordException.

def sum_x_y(data: dict) -> str:
    if "x" not in data and "y" not in data:
        raise MissingBothCoordException("Both x and y coord missing.")
    return data["x"] + data["y"]

And when we re-run the test, it passes.

test_example.py .                                          [100%]

======================= 1 passed in 0.01s ========================

Great! And that is pretty much it. This is how you check if an exception is raised withpytest. In the next section, we’re going to improve our function and we’ll need another test.

How to Assert the Exception Message - And Type

In this section, we’ll improve our sum_x_y function and also the tests. I’ll show you how you can make your test more robust by checking the exception message.

With that in mind, let’s expand the sum_x_y function.

def sum_x_y(data: dict) -> str:
    if "x" not in data and "y" not in data and "extra" not in data:
        raise MissingBothCoordException("Both X and Y coord missing.")
    if "x" not in data:
        raise MissingCoordException("The Y coordinate is not present in the data.")
    if "y" not in data:
        raise MissingCoordException("The Y coordinate is not present in the data.")
    return data["x"] + data["y"]

The new test goes like this:

def test_sum_x_y_has_x_missing_coord():
    data = {"extra": 1, "y": 2}
    with pytest.raises(MissingCoordException):
        sum_x_y(data)

And it passes!

$ poetry run pytest -k test_sum_x_y_has_x_missing_coord
====================== test session starts =======================
collected 2 items / 1 deselected / 1 selected                    

test_example.py .                                          [100%]

================ 1 passed, 1 deselected in 0.01s =================

However, it’s a bit fragile... In case you haven’t noticed it, when "x" is missing, the exception message is: "The Y coordinate is not present in the data.". This is a bug, and one way to detect it is by asserting we return the right message. Thankfully, pytest makes it easier to do.

If we refactor the test to take into account the message, we get the following output:

def test_sum_x_y_has_x_missing_coord():
    data = {"extra": 1, "y": 2}
    with pytest.raises(MissingCoordException) as exc:
        sum_x_y(data)
    assert "The X coordinate is not present in the data." in str(exc.value)

============================ FAILURES ============================
_____________ test_sum_x_y_has_x_missing_coord _____________

def test_sum_x_y_has_x_missing_coord():
        data = {"extra": 1, "y": 2}
        with pytest.raises(MissingCoordException) as exc:
            sum_x_y(data)
>       assert "The X coordinate is not present in the data." in str(exc.value)
E       AssertionError: assert 'The X coordinate is not present in the data.' in 'The Y coordinate is not present in the data.'
E        +  where 'The Y coordinate is not present in the data.' = str(MissingCoordException('The Y coordinate is not present in the data.'))
E        +    where MissingCoordException('The Y coordinate is not present in the data.') = <ExceptionInfo MissingCoordException('The Y coordinate is not present in the data.') tblen=2>.value

test_example.py:32: AssertionError
==================== short test summary info =====================
FAILED test_example.py::test_sum_x_y_has_x_missing_coord
======================= 1 failed in 0.02s ========================

That's exactly what we want. Let's fix the code and re-run the test.

def sum_x_y(data: dict) -> str:
    if "x" not in data and "y" not in data and "extra" not in data:
        raise MissingBothCoordException("Both X and Y coord missing.")
    if "x" not in data:
        raise MissingCoordException("The X coordinate is not present in the data.")
    if "y" not in data:
        raise MissingCoordException("The Y coordinate is not present in the data.")
    return data["x"] + data["y"]

And the result...

$ poetry run pytest test_example.py::test_sum_x_y_has_x_missing_coord
====================== test session starts =======================
platform linux -- Python 3.8.5, pytest-6.1.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/miguel/projects/tutorials/pytest-raises
collected 1 item                                                 

test_example.py .                                          [100%]

======================= 1 passed in 0.01s ========================

This is possible because pytest.raises returns an ExceptionInfo object that contains fields such as type, value, traceback and many others. If we wanted to assert the type, we could do something along these lines...

def test_sum_x_y_has_x_missing_coord():
    data = {"extra": 1, "y": 2}
    with pytest.raises(MissingCoordException) as exc:
        sum_x_y(data)
    assert "The X coordinate is not present in the data." in str(exc.value)
    assert exc.type == MissingCoordException

However, we are already asserting that by using pytest.raises so I think asserting the type like this a bit redundant. When is this useful then? It's useful if we are asserting a more generic exception in pytest.raises and we want to check the exact exception raised. For instance:

def test_sum_x_y_has_x_missing_coord():
    data = {"extra": 1, "y": 2}
    with pytest.raises(Exception) as exc:
        sum_x_y(data)
    assert "The X coordinate is not present in the data." in str(exc.value)
    assert exc.type == MissingCoordException

One more way to assert the message is by setting the match argument with the pattern you want to be asserted. The following example was taken from the official pytest docs.

>>> with raises(ValueError, match='must be 0 or None'):
...     raise ValueError("value must be 0 or None")

>>> with raises(ValueError, match=r'must be \d+$'):
...     raise ValueError("value must be 42")

As you can see, we can verify if the expected exception is raised but also if the message matches the regex pattern.

How to Assert That NO Exception Is Raised

The last section in this tutorial is about yet another common use case: how to assert that no exception is thrown. One way we can do that is by using a try / except. If it raises an exception, we catch it and assert False.

def test_sum_x_y_works():
    data = {"extra": 1, "y": 2, "x": 1}

    try:
        sum_x_y(data)
    except Exception as exc:
        assert False, f"'sum_x_y' raised an exception {exc}"

When we run this test, it passes.

$ poetry run pytest test_example.py::test_sum_x_y_works
====================== test session starts =======================
collected 1 item                                                 

test_example.py .                                          [100%]

======================= 1 passed in 0.00s ========================

Now, let's create a deliberate bug so we can see the test failing. We'll change our function to raise an ValueError before returning the result.

def sum_x_y(data: dict) -> str:
    if "x" not in data and "y" not in data and "extra" not in data:
        raise MissingBothCoordException("'extra field and x / y coord missing.")
    if "x" not in data:
        raise MissingCoordException("The X coordinate is not present in the data.")
    if "y" not in data:
        raise MissingCoordException("The Y coordinate is not present in the data.")
    raise ValueError("Oh no, this shouldn't have happened.")
    return data["x"] + data["y"]

And then we re-run the test...

    def test_sum_x_y_works():
        data = {"extra": 1, "y": 2, "x": 1}

        try:
            sum_x_y(data)
        except Exception as exc:
>           assert False, f"'sum_x_y' raised an exception {exc}"
E           AssertionError: 'sum_x_y' raised an exception Oh no, this shouldn't have happened.
E           assert False

test_example.py:52: AssertionError
==================== short test summary info =====================
FAILED test_example.py::test_sum_x_y_works - AssertionErr...
======================= 1 failed in 0.02s ========================

It works! Our code raised the ValueError and the test failed!

Conclusion

That’s it for today, folks! I hope you’ve learned something new and useful. Knowing how to test exceptions is an important skill to have. The way pytest does that is, IMHO, cleaner than unittest and much less verbose. In this article, I showed how you can not only assert that your code raises the expected exception, but also assert when they’re not supposed to be raised. Finally, we saw how to check if the exception message is what you expect, which makes test cases more reliable.

Other posts you may like:

How to Check if an Exception Is Raised (or Not) With pytest first appeared on miguendes's blog.

See you next time!

How to Pass Multiple Arguments to a map Function in Python

Miguel Brito — Thu, 09 Sep 2021 07:40:28 +0000

Introduction

In this post, I’m going to show what you can do to map a function that expects multiple arguments.

There are multiple ways of doing that and they work not only for the regular map function, you can also use the trick to pass multiple parameters to Executor.map from the concurrent.futures and multiprocessing.Pool.

The Problem With the map() Function

1.1. Solution 1 - Mapping Multiple Arguments with itertools.starmap()

1.2. Solution 2 - Using functools.partial to “Freeze” the Arguments

1.3. Solution 3 - Mapping Multiple Arguments by "Repeating" Them
Problem 2: Passing Multiple Parameters to multiprocessing Pool.map

2.1. Using pool.starmap

2.2. Using partial()
Problem 3: How to Pass Multiple Arguments to a concurrent futures ProcessPoolExecutor (or ThreadPoolExecutor)?

3.1. Using partial()

3.2. Using repeat()
Conclusion

The Problem With the `map()` Function

Let’s imagine that you have a function called sum_four that takes 4 arguments and returns their sum.

>>> def sum_four(a, b, c, d):
        return a + b + c + d

Let’s also suppose that you are solving a very specific problem that requires the first 3 arguments to be fixed. In this problem, you want to compare how the function behaves when you vary only the last parameter.

>>> a, b, c = 1, 2, 3

>>> sum_four(a=a, b=b, c=c, d=1)
 7

>>> sum_four(a=a, b=b, c=c, d=2)
 8

>>> sum_four(a=a, b=b, c=c, d=3)
 9

>>> sum_four(a=a, b=b, c=c, d=4)
 10

Now, say that you want to use map, because you like functional programming, or maybe because you come from a language that encourages this paradigm.

Since only d varies, we could store all potential values for d we want to test in a list like this all_d_values = [1, 2, 3, 4].

The issue is, given a function and a list of single elements, if you want to pass that list to a map function and it takes only one element, what can you do?

Solution 1 - Mapping Multiple Arguments with `itertools.starmap()`

The first solution is to not adopt the map function but use itertools.starmap instead. This function will take a function as arguments and an iterable of tuples. Then, starmap will iterate over each tuple t and call the function by unpacking the arguments, like this for t in tuples: function(*t).

To make things more clear, consider the following example.

>>> import itertools

>>> all_d_values = [1, 2, 3, 4]

>>> items = ((a, b, c, d) for d in all_d_values)

>>> list(items)
 [(1, 2, 3, 1), (1, 2, 3, 2), (1, 2, 3, 3), (1, 2, 3, 4)]

>>> list(itertools.starmap(sum_four, items))
 [7, 8, 9, 10]

As you can see, there’s a lot of repetition, which may inevitably consume a lot of memory if the list is big. To improve that I made items as a generator, this way we only hold in memory the element we’ll be processing.

Solution 2 - Using `functools.partial` to “Freeze” the Arguments

The second solution is to use currying and create a new partial function. According to the docs, partial() will "freeze" some portion of a function’s arguments and/or keywords resulting in a new function with a simplified signature.

>>> import functools

>>> partial_sum_four = functools.partial(sum_four, a, b, c)

>>> partial_sum_four(3)
9

>>> list(map(partial_sum_four, all_d_values))
[7, 8, 9, 10]

Solution 3 - Mapping Multiple Arguments by "Repeating" Them

The third alternative is to use the itertools.repeat().

This function produces an iterator that returns object over and over again. It will run indefinitely if you don’t specify the times argument.

If we take a closer look at map()'s signature, it accepts a function and multiple iterables, map(function, iterable, ...).

According to its description,

If additional iterable arguments are passed, function must take that many arguments and is applied to the items from all iterables in parallel. With multiple iterables, the iterator stops when the shortest iterable is exhausted.

Bingo! We can make a, b and c infitnite iterables by using itertools.repeat(). As soon as all_d_values is exhausted, which is the shortest iterable, map() will stop.

>>> import itertools
>>> list(map(sum_four, itertools.repeat(a), itertools.repeat(b), itertools.repeat(c), all_d_values))
 [7, 8, 9, 10]

To put it another way, using repeat() is roughly equivalent to:

>>> list(map(sum_four, [1, 1, 1, 1], [2, 2, 2, 2], [3, 3, 3, 3], all_d_values))
 [7, 8, 9, 10]

You don't need to worry too much about memory as repeat produces the elements on the go. In fact, it returns a repeatobject, not list [ref] .

Problem 2: Passing Multiple Parameters to multiprocessing `Pool.map`

This problem is very similar to using the regular map(). The only difference is that we need to pass multiple arguments to the multiprocessing's pool map.

Suppose that we want to speed up our code and run sum_four in parallel using processes.

The good news is, you can use the solutions above, with one exception: Pool.map only accepts one iterable. This means we cannot use repeat() here. Let's see the alternatives.

Using `pool.starmap`

The Pool class from multiprocessing module implements a starmap function that works the same way as its counterpart from the itertools module.

>>> from multiprocessing import Pool

>>> import itertools

>>> def sum_four(a, b, c, d):
                return a + b + c + d

>>> a, b, c = 1, 2, 3

>>> all_d_values = [1, 2, 3, 4]

>>> items = [(a, b, c, d) for d in all_d_values]

>>> items
 [(1, 2, 3, 1), (1, 2, 3, 2), (1, 2, 3, 3), (1, 2, 3, 4)]

>>> with Pool(processes=4) as pool:
         res = pool.starmap(sum_four, items)

>>> res
 [7, 8, 9, 10]

Using `partial()`

As alternative, we can also rely on the good partial function.


>>> import functools

>>> partial_sum_four = functools.partial(sum_four, a, b, c)

>>> with Pool(processes=4) as pool:
         res = pool.map(partial_sum_four, all_d_values)

>>> res
 [7, 8, 9, 10]

Problem 3: How to Pass Multiple Arguments to concurrent futures `Executor.map`?

The concurrent.futures module provides a high-level interface called Executor to run callables asynchronously.

There are two different implementations available, a ThreadPoolExecutor and a ProcessPoolExecutor.

Contrary to multiprocessing.Pool, a Executor does not have a startmap() function. However, its map() implementation supports multiple iterables, which allow us to use repeat(). Another difference is that Executor.map returns a generator, not a list.

Using `partial()` With a ProcessPoolExecutor (or ThreadPoolExecutor)

By "freezing" the arguments using partial we use the map method from ProcessPoolExecutor like a regular map function. Since they both share the same interface, you can do the same interchangeably with a ThreadPoolExecutor

>>> from concurrent.futures import ProcessPoolExecutor

>>> import functools

>>> def sum_four(a, b, c, d):
                return a + b + c + d

>>> a, b, c = 1, 2, 3

>>> all_d_values = [1, 2, 3, 4]

>>> partial_sum_four = functools.partial(sum_four, a, b, c)

>>> with ProcessPoolExecutor(max_workers=4) as pool:
              res = list(pool.map(partial_sum_four, all_d_values))

>>> res
 [7, 8, 9, 10]

Using `repeat()`

Again, we can just use itertools.repeat to get the job done like the previous solutions.

>>> from concurrent.futures import ProcessPoolExecutor

>>> from itertools import repeat

>>> def sum_four(a, b, c, d):
                return a + b + c + d

>>> a, b, c = 1, 2, 3

>>> all_d_values = [1, 2, 3, 4]

>>> with ProcessPoolExecutor(max_workers=4) as pool:
              res = list(pool.map(sum_four, repeat(a), repeat(b), repeat(c), all_d_values))

>>> res
 [7, 8, 9, 10]

Conclusion

That’s it for today, folks! I hope you’ve learned something different and useful. The map() function makes Python feel like a functional programming language. map() is available not only as a built-in function but also as methods in the multiprocessing and concurrent.futures module. In this article, I showed what I do to map functions that take several arguments.

Other posts you may like:

This post was originally published at https://miguendes.me/how-to-pass-multiple-arguments-to-a-map-function-in-python

See you next time!

What if Python Had This Ruby Feature?

Miguel Brito — Sat, 21 Aug 2021 09:15:00 +0000

In this post, I'll present how I changed Python to accept "else-less" if expressions, similar to Ruby's "inline if", also known as conditional modifier 👇

Why?

The idea of having an else-less if expression in Python came to my mind when I had to work with a Ruby service at my past job. Ruby, contrary to Python, makes lots of things implicit [Citation needed], and this kind of if expression is one of them. I say it's implicit because it returns nil if the expression evaluates to false. This is also called conditional modifier.

$ irb
irb(main):001:0> RUBY_VERSION
=> "2.7.1"
irb(main):002:0> a = 42 if true
=> 42
irb(main):003:0> b = 21 if false
=> nil
irb(main):004:0> b
=> nil
irb(main):005:0> a
=> 42

In Python, one cannot do that without explicitly adding an else to the expression. In fact, as of this PR, the interpreter will tell right away that the else is mandatory in the SyntaxError message.

$ ./python
Python 3.11.0a0 (heads/main:938e84b4fa, Aug  6 2021, 08:59:36) [GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> a = 42 if True
  File "<stdin>", line 1
    a = 42 if True
        ^^^^^^^^^^
SyntaxError: expected 'else' after 'if' expression

However, I find Ruby's if actually very convenient. This convenience became more evident when I had to go back to Python and write things like this:

>>> my_var = 42 if some_cond else None

So I thought to myself, what would be like if Python had similar feature? Could I do it myself? How hard would that be?

How?

Me trying to make sense of CPython's source code.

Digging into CPython's code and changing the language's syntax sounded not trivial to me.

Luckily, during the same week, I found out on Twitter that Anthony Shaw had just written a book on CPython Internals and it was available for pre-release. I didn't think twice and bought the book.

I've got to be honest, I'm the kind of person who buys things and doesn't use them immediately. As I had other plans in mind, I let it "getting dust" in my home folder for a while.

Until... I had to work with that Ruby service again. It reminded me of the CPython Internals book and how challenging hacking the guts of Python would be.

First thing was to go though the book from the very start and try to follow each step. The book focus on Python 3.9, so in order to follow though it, one needs to checkout the 3.9 tag, and that's what I did.

I learned about how the code is structured and then how to compile it. The next chapters show how to extend the grammar and add new things, such as a new operator.

As I got familiar with the code base and how to tweak the grammar, I decided to give it a spin and make my own changes to it.

The First (Failed) Attempt

As I started finding my way around CPython's code from the latest main branch, I noticed that lots of things had changed since Python 3.9, yet some fundamental concepts didn't.

My first shot was to dig into the grammar definition and find the if expression rule. The file is currently named Grammar/python.gram. Locating it was not difficult, an ordinary CTRL+F for the 'else' keyword was enough.

file: Grammar/python.gram
...
expression[expr_ty] (memo):
    | invalid_expression
    | a=disjunction 'if' b=disjunction 'else' c=expression { _PyAST_IfExp(b, a, c, EXTRA) }
    | disjunction
    | lambdef
....

Now with the rule in hand, my idea was to add one more option to the current if expression where it would match a=disjunction 'if' b=disjunction and c expression would be NULL.

This new rule should be placed immediately after the complete one, otherwise the parser would match a=disjunction 'if' b=disjunction always, returning a SyntaxError.

expression[expr_ty] (memo):
    | invalid_expression
    | a=disjunction 'if' b=disjunction 'else' c=expression { _PyAST_IfExp(b, a, c, EXTRA) }
    | a=disjunction 'if' b=disjunction  { _PyAST_IfExp(b, a, NULL, EXTRA) }
    | disjunction
    | lambdef
....

Regenerating the Parser and Compiling Python From Source

CPython comes with a Makefile containing lots of useful commands. One of them is the regen-pegen command which converts Grammar/python.gram into Parser/parser.c.

Besides changing the grammar, I had to modify the AST for the if expression. AST stands for Abstract Syntax Tree and it is a way of representing the syntactic structure of the grammar as a tree. For a more information about ASTs, I highly recommend the Crafting Interpreters book by Robert Nystrom.

Moving on, if you observe the rule for if expression goes like this:

    | a=disjunction 'if' b=disjunction 'else' c=expression { _PyAST_IfExp(b, a, c, EXTRA) }

The means, when the parser finds this rule, it calls the _PyAST_IfExp which gives back a expr_ty data structure. So this gave me a clue, in order to implement the behavior of the new rule, I'd need to change _PyAST_IfExp.

To find where is located, I used my rip-grep skills and searched for it inside the source root.

$ rg _PyAST_IfExp -C2 .

[OMITTED]
Python/Python-ast.c
2686-
2687-expr_ty
2688:_PyAST_IfExp(expr_ty test, expr_ty body, expr_ty orelse, int lineno, int
2689-             col_offset, int end_lineno, int end_col_offset, PyArena *arena)
2690-{
[OMITTED]

... And the implementation goes like this:

expr_ty
_PyAST_IfExp(expr_ty test, expr_ty body, expr_ty orelse, int lineno, int
             col_offset, int end_lineno, int end_col_offset, PyArena *arena)
{
    expr_ty p;
    if (!test) {
        PyErr_SetString(PyExc_ValueError,
                        "field 'test' is required for IfExp");
        return NULL;
    }
    if (!body) {
        PyErr_SetString(PyExc_ValueError,
                        "field 'body' is required for IfExp");
        return NULL;
    }
    if (!orelse) {
        PyErr_SetString(PyExc_ValueError,
                        "field 'orelse' is required for IfExp");
        return NULL;
    }
    p = (expr_ty)_PyArena_Malloc(arena, sizeof(*p));
    if (!p)
        return NULL;
    p->kind = IfExp_kind;
    p->v.IfExp.test = test;
    p->v.IfExp.body = body;
    p->v.IfExp.orelse = orelse;
    p->lineno = lineno;
    p->col_offset = col_offset;
    p->end_lineno = end_lineno;
    p->end_col_offset = end_col_offset;
    return p;
}

Since I pass orelse as NULL, I thought it was just a matter of changing the body of if (!orelse) and assign None to orelse.

    if (!orelse) {
-       PyErr_SetString(PyExc_ValueError,
-                       "field 'orelse' is required for IfExp");
-       return NULL;
+       orelse = Py_None;
    }

Now time to test it, I compile the code with make -j8 -s and fire up the interpreter.

$ make -j8 -s                                                                                                                                                            
Python/Python-ast.c: In function ‘_PyAST_IfExp’:
Python/Python-ast.c:2703:16: warning: assignment from incompatible pointer type [-Wincompatible-pointer-types]
         orelse = Py_None;

Despite the glaring obvious warnings, I decided to ignore it just to see what happens. 😅

$ ./python
Python 3.11.0a0 (heads/ruby-if-new-dirty:f92b9133ef, Aug  2 2021, 09:13:02) [GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> a = 42 if True
>>> a
42
>>> b = 21 if False
[1]    16805 segmentation fault (core dumped)  ./python

Ouch! It works for the if True case, but assigning Py_None to expr_ty orelse causes a segfault.

Time to go back to see what is wrong.

The Second Attempt

It wasn't too difficult to figure out where I messed up. orelse is a expr_ty and I'm assigning to it a Py_None which is a PyObject *. Again, thanks to rip-grep I found its definition.

$ rg constant -tc -C2

Include/internal/pycore_asdl.h
14-typedef PyObject * string;
15-typedef PyObject * object;
16:typedef PyObject * constant;

Now, how did I find out Py_None was a constant?

Whilst reviewing Grammar/python.gram file, I found that one of the rules for the new pattern matching syntax is defined like this:

# Literal patterns are used for equality and identity constraints
literal_pattern[pattern_ty]:
    | value=signed_number !('+' | '-') { _PyAST_MatchValue(value, EXTRA) }
    | value=complex_number { _PyAST_MatchValue(value, EXTRA) }
    | value=strings { _PyAST_MatchValue(value, EXTRA) }
    | 'None' { _PyAST_MatchSingleton(Py_None, EXTRA) }

However, this rule is a pattern_ty not an expr_ty. But that's fine. What really matters is to understand what _PyAST_MatchSingleton actually is. Then, I searched for it in Python/Python-ast.c.

file: Python/Python-ast.c
...
pattern_ty
_PyAST_MatchSingleton(constant value, int lineno, int col_offset, int
                      end_lineno, int end_col_offset, PyArena *arena)
...

Now back to the "drawing board", I look for the definition of a None node in the grammar. To my great relief, I find it!

atom[expr_ty]:
    | NAME
    | 'True' { _PyAST_Constant(Py_True, NULL, EXTRA) }
    | 'False' { _PyAST_Constant(Py_False, NULL, EXTRA) }
    | 'None' { _PyAST_Constant(Py_None, NULL, EXTRA) }
....

At this point, I had all the information I needed. To return a expr_ty representing None I need to create a node in the AST which is constant by using the _PyAST_Constant function.

    | a=disjunction 'if' b=disjunction 'else' c=expression { _PyAST_IfExp(b, a, c, EXTRA) }
-   | a=disjunction 'if' b=disjunction { _PyAST_IfExp(b, a, NULL, EXTRA) }
+   | a=disjunction 'if' b=disjunction { _PyAST_IfExp(b, a, _PyAST_Constant(Py_None, NULL, EXTRA), EXTRA) }
    | disjunction

Now I must revert Python/Python-ast.c as well. Since I'm feeding it a valid expr_ty, it will never be NULL.

file: Python/Python-ast.c
...
     if (!orelse) {
-        orelse = Py_None;
+        PyErr_SetString(PyExc_ValueError,
+                        "field 'orelse' is required for IfExp");
+        return NULL;
     }
...

Let's compile it again and see what happens!

$ make -j8 -s && ./python                             
Python 3.11.0a0 (heads/ruby-if-new-dirty:25c439ebef, Aug  2 2021, 09:25:18) [GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> c = 42 if True
>>> c
42
>>> b = 21 if False
>>> type(b)
<class 'NoneType'>
>>>

WOT!? It works! 🎉🎉🎉

Now, we need to do one more test. Ruby functions allow returning a value if a condition matches and if not, the rest of the function body gets executed. Like this 👇

At this point I wonder if that would work out-of-the-box. I rush to the interpreter again and write the same function.

>>> def f(test):
...     return 42 if test
...     print('missed return')
...     return 21
... 
>>> f(False)
>>> f(True)
42
>>>

Ooopss...

The function returns None if test is False... To help me debug this, I summoned the ast module. The official docs define it like so:

The ast module helps Python applications to process trees of the Python abstract syntax grammar. The abstract syntax itself might change with each Python release; this module helps to find out programmatically what the current grammar looks like.

Let's print the AST for this function...

>>> fc = '''
... def f(test):
...     return 42 if test
...     print('missed return')
...     return 21
... '''
>>> print(ast.dump(ast.parse(fc), indent=4))
Module(
    body=[
        FunctionDef(
            name='f',
            args=arguments(
                posonlyargs=[],
                args=[
                    arg(arg='test')],
                kwonlyargs=[],
                kw_defaults=[],
                defaults=[]),
            body=[
                Return(
                    value=IfExp(
                        test=Name(id='test', ctx=Load()),
                        body=Constant(value=42),
                        orelse=Constant(value=None))),
                Expr(
                    value=Call(
                        func=Name(id='print', ctx=Load()),
                        args=[
                            Constant(value='missed return')],
                        keywords=[])),
                Return(
                    value=Constant(value=21))],
            decorator_list=[])],
    type_ignores=[])

Now things make more sense, my change to the grammar was just a syntax sugar. It turns an expression like this a if b into this a if b else None. The problem here is that Python will return no matter what, so the rest of the function is ignored.

We can also look at the bytecode generated to understand what exactly is executed by the interpreter. And for that, we can use the dis module. According to the docs:

The dis module supports the analysis of CPython bytecode by disassembling it.

>>> import dis
>>> dis.dis(f)
  2           0 LOAD_FAST                0 (test)
              2 POP_JUMP_IF_FALSE        4 (to 8)
              4 LOAD_CONST               1 (42)
              6 RETURN_VALUE
        >>    8 LOAD_CONST               0 (None)
             10 RETURN_VALUE

What this basically means is that in case the test is false, the execution jumps to 8, which loads the None into the top of the stack and returns it.

Supporting "return-if"

To support the same Ruby feature, I can turn the expression return 42 if test into a regular if statement that returns if test is true.

To do that, I needed to add one more rule. This time, it would be a rule that matches the return <value> if <test> piece of code. Not only that, we need a _PyAST_ function that creates the node for us. Let's then call it _PyAST_ReturnIfExpr.

file: Grammar/python.gram

return_stmt[stmt_ty]:
+   | 'return' a=star_expressions 'if' b=disjunction { _PyAST_ReturnIfExpr(a, b, EXTRA) }
    | 'return' a=[star_expressions] { _PyAST_Return(a, EXTRA) }

As mentioned previously, the implementation for all these functions reside in Python/Python-ast.c, and the their definition in Include/internal/pycore_ast.h, so I put _PyAST_ReturnIfExpr there.

file: Include/internal/pycore_ast.h

 stmt_ty _PyAST_Return(expr_ty value, int lineno, int col_offset, int
                       end_lineno, int end_col_offset, PyArena *arena);
+stmt_ty _PyAST_ReturnIfExpr(expr_ty value, expr_ty test, int lineno, int col_of
fset, int
+                      end_lineno, int end_col_offset, PyArena *arena);
 stmt_ty _PyAST_Delete(asdl_expr_seq * targets, int lineno, int col_offset, int
                       end_lineno, int end_col_offset, PyArena *arena);

file: Python/Python-ast.c

 }

+stmt_ty
+_PyAST_ReturnIfExpr(expr_ty value, expr_ty test, int lineno, int col_offset, int end_lineno, int
+              end_col_offset, PyArena *arena)
+{
+    stmt_ty ret, p;
+    ret = _PyAST_Return(value, lineno, col_offset, end_lineno, end_col_offset, arena);
+
+    asdl_stmt_seq *body;  
+    body = _Py_asdl_stmt_seq_new(1, arena); 
+    asdl_seq_SET(body, 0, ret);
+
+    p = _PyAST_If(test, body, NULL, lineno, col_offset, end_lineno, end_col_offset, arena);
+
+    return p;
+}
+
 stmt_ty

Let's pause for a bit to examine the implementation of _PyAST_ReturnIfExpr. Like I mentioned previously, I want to turn return <value> if <test> into if <test>: return <value>.

Both return and the regular if are statements, so in CPython they're represented as stmt_ty. The _PyAST_If expectes a expr_ty test and a body, which is a sequence of statements. In this case, body is asdl_stmt_seq *body.

As a result, what we really want here is a if statement with a body where the only statement is a return <value> one.

CPython disposes of some convenient functions to build asdl_stmt_seq * and one of them is _Py_asdl_stmt_seq_new. So I used it to create the body and add the return statement I created a few lines before with _PyAST_Return.

Once that's done, the last step is to pass the test as well as the body to _PyAST_If.

And before I forget, you may be wondering what on earth is the PyArena *arena. Arena is a CPython abstraction used for memory allocation. It allows efficient memory usage by using memory mapping mmap() and placing them in contiguous chunks of memory [reference].

Now it's time to regenerate the parser and test it one more time.

>>> def f(test):
...     return 42 if test
...     print('missed return')
...     return 21
... 
>>> import dis
>>> f(False)
>>> f(True)
42

Oh no! It doesn't work...

Let's check the bytecodes...


>>> dis.dis(f)
  2           0 LOAD_FAST                0 (test)
              2 POP_JUMP_IF_FALSE        4 (to 8)
              4 LOAD_CONST               1 (42)
              6 RETURN_VALUE
        >>    8 LOAD_CONST               0 (None)
             10 RETURN_VALUE
>>>

... the same bloody bytecode instructions again!

Going Back to the Compilers Class

At that point, I was clueless. I had no idea what was going on until... I decided to go down the rabbit hole of expanding the grammar rules.

The new rule I added went like this 'return' a=star_expressions 'if' b=disjunction { _PyAST_ReturnIfExpr(a, b, EXTRA) }.

My only hypothesis is that a=star_expressions 'if' b=disjunction is being resolved to the else-less rule I added in the beginning.

By going over the grammar one more time, I figure that my theory holds. star_expressions will match a=disjunction 'if' b=disjunction { _PyAST_IfExp(b, a, NULL, EXTRA) }.

The only way to fix this is by getting rid of the star_expressions. So I change the rule to:

 return_stmt[stmt_ty]:
-    | 'return' a=star_expressions 'if' b=disjunction { _PyAST_ReturnIfExpr(a, b, EXTRA) }
+    | 'return' a=disjunction guard=guard !'else' { _PyAST_ReturnIfExpr(a, guard, EXTRA) }
     | 'return' a=[star_expressions] { _PyAST_Return(a, EXTRA) }

You might be wondering, what is guard and what is !else and what is star_expressions?

This 'guard' is a rule that is part of the pattern matching rules. The new pattern matching feature added in Python 3.10 allows things like this:

match point:
    case Point(x, y) if x == y:
        print(f"Y=X at {x}")
    case Point(x, y):
        print(f"Not on the diagonal")

And the rule goes by this:

guard[expr_ty]: 'if' guard=named_expression { guard }

With that, I added one more check. To avoid it failing with SyntaxError, we need to make sure the rule matches only code like this: return value if cond. Thus, to prevent code such as return an if cond else b being matched prematurely, I added a !'else' to the rule.

Last, but not least, the star_expressions allow us to return to return destructured iterables. For example:

>>> def f():
   ...:     a = [1, 2]
   ...:     return 0, *a
   ...: 

>>> f()
(0, 1, 2)

In this case, 0, *a is a tuple, which falls under the category of star_expressions. The regular if-expression doesn't allow using star_expressions with it AFAIK, so changing our new return rule won't be an issue.

Does it work yet?

After fixing the return rule, I regenerate the grammar one more time and compile it.

>>> def f(test):
...     return 42 if test
...     print('missed return')
...     return 21
... 
>>> f(False)
missed return
21
>>> f(True)
42

And... IT WORKS!!

Let's check the bytecode then...

>>> import dis
>>> dis.dis(f)
  2           0 LOAD_FAST                0 (test)
              2 POP_JUMP_IF_FALSE        4 (to 8)
              4 LOAD_CONST               1 (42)
              6 RETURN_VALUE

  3     >>    8 LOAD_GLOBAL              0 (print)
             10 LOAD_CONST               2 ('missed return')
             12 CALL_FUNCTION            1
             14 POP_TOP

  4          16 LOAD_CONST               3 (21)
             18 RETURN_VALUE
>>>

That's precisely what I wanted. In fact, to make sure, let also see if the AST is the same as the one with regular if.

>>> import ast
>>> print(ast.dump(ast.parse(fc), indent=4))
Module(
    body=[
        FunctionDef(
            name='f',
            args=arguments(
                posonlyargs=[],
                args=[
                    arg(arg='test')],
                kwonlyargs=[],
                kw_defaults=[],
                defaults=[]),
            body=[
                If(
                    test=Name(id='test', ctx=Load()),
                    body=[
                        Return(
                            value=Constant(value=42))],
                    orelse=[]),
                Expr(
                    value=Call(
                        func=Name(id='print', ctx=Load()),
                        args=[
                            Constant(value='missed return')],
                        keywords=[])),
                Return(
                    value=Constant(value=21))],
            decorator_list=[])],
    type_ignores=[])
>>>

And indeed it is!

If(
    test=Name(id='test', ctx=Load()),
    body=[
        Return(
            value=Constant(value=42))],
     orelse=[]),

This node is the same as the one that would be generated by

if test: return 42

If It's Not Tested, It's Broken?

To conclude this journey, I thought it'd be a good idea to add some unit tests as well. Before writing anything new, I wanted to get an idea of what I had broken.

With the code tested manually, I run all tests using the 'test' module, python -m test -j8. The -j8 means we'll use 8 processes to run the tests in parallel.

$ ./python -m test -j8

To my surprise, only one test fail! 😱

== Tests result: FAILURE ==

406 tests OK.

1 test failed:
    test_grammar

Since I ran all tests, it's hard to navigate on the output so I can run only this one again in isolation.

======================================================================
FAIL: test_listcomps (test.test_grammar.GrammarTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/miguel/projects/cpython/Lib/test/test_grammar.py", line 1732, in test_listcomps
    check_syntax_error(self, "[x if y]")
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miguel/projects/cpython/Lib/test/support/__init__.py", line 497, in check_syntax_error
    with testcase.assertRaisesRegex(SyntaxError, errtext) as cm:
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: SyntaxError not raised

----------------------------------------------------------------------

Ran 76 tests in 0.038s

FAILED (failures=1)
test test_grammar failed
test_grammar failed (1 failure)

== Tests result: FAILURE ==

1 test failed:
    test_grammar

1 re-run test:
    test_grammar

Total duration: 82 ms
Tests result: FAILURE

And there it is! It expects a syntax error when running a [x if y] expression. We can safely remove it and re-run the tests again.

== Tests result: SUCCESS ==

1 test OK.

Total duration: 112 ms
Tests result: SUCCESS

Now that everything is OK, it's time to add a few more tests. It's important to test not only the new "else-less if" but also the new return statement.

By navigating though the test_grammar.py file we can find a test for pretty much every grammar rule. The first one I look for is test_if_else_expr. This test doesn't fail, so it only tests for the happy case. To make it more robust we need to add two new tests to check if True and if False case.

        self.assertEqual((6 < 4 if 0), None)
        self.assertEqual((6 < 4 if 1), False)

I run everything again, all tests pass this time.

ps: bool in Python is subclass of integer, so you can use 1 to denote True and 0 for False

Ran 76 tests in 0.087s

OK

== Tests result: SUCCESS ==

1 test OK.

Total duration: 174 ms
Tests result: SUCCESS

Lastly, we need the tests for the return rule. They're defined in the test_return test. Just like the if expression one, this test pass with no modification.

To test this new use case, I create a function that receives a bool argument and returns if the argument is true, when it's false, it skips the return, just like the manual tests I have been doing up to this point.

        def g4(test):
            a = 1
            return a if test
            a += 1
            return a

        self.assertEqual(g4(False), 2)
        self.assertEqual(g4(True), 1)

Now, save the file and re-run test_grammar one more time.

----------------------------------------------------------------------

Ran 76 tests in 0.087s

OK

== Tests result: SUCCESS ==

1 test OK.

Total duration: 174 ms
Tests result: SUCCESS

All good in the hood! test_grammar passes with flying colors and the last thing, just in case, is to re-run the full test suite.

$ ./python -m test -j8

After a while, all tests pass and I'm very happy with the result.

Limitations

If you know Ruby well, by this point you've probably noticed that what I did here is not 100% the same as a conditional modifier. For example, in Ruby you can run actual expressions in these modifiers.

irb(main):002:0> a = 42
irb(main):003:0> a += 1 if false
=> nil
irb(main):004:0> a
=> 42
irb(main):005:0> a += 1 if true
=> 43

I cannot do the same with my implementation.

>>> a = 42
>>> a += 1 if False
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for +=: 'int' and 'NoneType'
>>> a += 1 if True
>>> a
43

What this reveals is that the return rule I created is just a workaround. If I want to make it as close as possible to Ruby's conditional modifier, I'll need to make it work with other statements as well, not just return.

Nevertheless, this is fine. My goal with this experiment was just to learn more about Python internals and see how would I navigate a little-known code base written in C and make the appropriate changes to it. And I have to admit that I'm pretty happy with the results!

Conclusion

Adding a new syntax inspired by Ruby is a really nice exercise to learn more about the internals of Python. Of course, if I had to convert this as a PR, the core developers would probably find a few shortcomings, as I have already found and described in the previous section. However, since I did this just for fun, I'm very happy with the results.

The source code with all my changes is on my CPython fork under the branch ruby-if-new.

I hope you've found this post cool and let's see what comes next!

Other posts you may like:

11 Useful Resources To Learn Python's Internals From Scratch

See you next time!

This post was originally published at https://miguendes.me/what-if-python-had-this-ruby-feature

Design Patterns That Make Sense in Python: Simple Factory

Miguel Brito — Sat, 31 Jul 2021 14:48:14 +0000

In the first post of this series, I'll talk about Design Patterns that make sense in Python. We'll see how to implement them and how they are used in the standard library and other third-party packages. In this post, we'll go over the Simple Factory Pattern and understand why it makes sense in Python.

Introduction

Design Patterns has been a popular subject since the Design Patterns: Elements of Reusable Object-Oriented Software (a.k.a GoF) book was released back in 1994. GoF’s goals were to show techniques, a.k.a patterns, to improve an object-oriented design. In total, the book demonstrated 23 patterns, classified in 3 groups:

Creational
Behavioral
Structural

Among the creational patterns, we have the Factory Method. According to the book, the goal of this pattern is to define an interface to create an object. The sub classes will then decide which class will be instantiated. There’s also another variation called Simple Factory. This pattern creates an instance of an object without exposing the details behind the construction. In this article, we’ll see how to do that in Python in an idiomatic way.

When this is useful? Can we just call the constructor directly?

This pattern is helpful when you need to perform an extra setup before calling a constructor. In the next section we’ll see several examples on how they are used in the Python standard library and also in third-party packages such as pandas.

Usage

In this part, we’ll see how this pattern is used in practice and how you can implement it yourself.

Python Standard Library

The datetime module is one of the most important ones in the standard library. It defines a few classes such as date, datetime, and timedelta. This module uses the simple factory pattern extensively. A real example is the date class. It has a method called fromtimestamp that creates date instances given a timestamp.

In [3]: from datetime import date

In [4]: date.fromtimestamp(time.time())
Out[4]: datetime.date(2020, 11, 10)

If we look at the implementation, we can see that it extracts the year, month and day from the time instance and the call the constructor (cls). This is the kind of setup that is abstracted away from the user.

    @classmethod
    def fromtimestamp(cls, t):
        "Construct a date from a POSIX timestamp (like time.time())."
        y, m, d, hh, mm, ss, weekday, jday, dst = _time.localtime(t)
        return cls(y, m, d)

Another great example is the fromisocalendar method, which performs an extensive setup. Instead of leaving it to the user, the class provides the functionality “for free” by hiding that from you.

# https://github.com/python/cpython/blob/c304c9a7efa8751b5bc7526fa95cd5f30aac2b92/Lib/datetime.py#L860-L893
...
    @classmethod
    def fromisocalendar(cls, year, week, day):
        """Construct a date from the ISO year, week number and weekday.
        This is the inverse of the date.isocalendar() function"""
        # Year is bounded this way because 9999-12-31 is (9999, 52, 5)
        if not MINYEAR <= year <= MAXYEAR:
            raise ValueError(f"Year is out of range: {year}")

        if not 0 < week < 53:
            out_of_range = True

            if week == 53:
                # ISO years have 53 weeks in them on years starting with a
                # Thursday and leap years starting on a Wednesday
                first_weekday = _ymd2ord(year, 1, 1) % 7
                if (first_weekday == 4 or (first_weekday == 3 and
                                           _is_leap(year))):
                    out_of_range = False

            if out_of_range:
                raise ValueError(f"Invalid week: {week}")

        if not 0 < day < 8:
            raise ValueError(f"Invalid weekday: {day} (range is [1, 7])")

        # Now compute the offset from (Y, 1, 1) in days:
        day_offset = (week - 1) * 7 + (day - 1)

        # Calculate the ordinal day for monday, week 1
        day_1 = _isoweek1monday(year)
        ord_day = day_1 + day_offset

        return cls(*_ord2ymd(ord_day))
....

Pandas

pandas is one of the most used Python packages thanks to the rise of Data Science and Machine Learning. Just like Python, pandas also makes use of factory methods. A classic example is the from_dict method that belongs to the DataFrame class.

        >>> data = {'row_1': [3, 2, 1, 0], 'row_2': ['a', 'b', 'c', 'd']}
        >>> pd.DataFrame.from_dict(data, orient='index')
               0  1  2  3
        row_1  3  2  1  0
        row_2  a  b  c  d

When we inspect the implementation we can also see a lot of setup and extra checks.

    @classmethod
    def from_dict(cls, data, orient="columns", dtype=None, columns=None) -> DataFrame:
        ...
        index = None
        orient = orient.lower()
        if orient == "index":
            if len(data) > 0:
                # TODO speed up Series case
                if isinstance(list(data.values())[0], (Series, dict)):
                    data = _from_nested_dict(data)
                else:
                    data, index = list(data.values()), list(data.keys())
        elif orient == "columns":
            if columns is not None:
                raise ValueError("cannot use columns parameter with orient='columns'")
        else:  # pragma: no cover
            raise ValueError("only recognize index or columns for orient")

        return cls(data, index=index, columns=columns, dtype=dtype)

How to Implement It

The most idiomatic way of implementing factory methods in Python is by decorating them as classmethod. In Python, regular methods are attached to an object instance. We can access the objects’ fields via the self argument. classmethod, on the other hand, are bound not to an instance but to a class. That means when we call MyClass.factory_method we are passing MyClass as the first argument, called cls. This property makes them an excellent alternative for factory methods since calling cls(args) inside a classmethod is the same as MyClass(args).

To design your own factory methods, it’s sufficient to decorate it as a classmethod and return a new instance built with the cls argument. For example, presume that we want to implement a Point class and we want it also be constructed from Polar coordinates. The extra setup to convert from Polar to Cartesian is kept inside the method. Not simply it’s more readable, but also simplifies the constructor.

class Point:
    def __init__(self, x: float, y: float):
        self.x = x
        self.y = y

    @classmethod
    def from_polar(cls, r: float, theta: float) -> "Point":
        """
        Converts a polar coordinate into cartesian point.

        >>> Point.from_polar(r=-2**0.5, theta=math.pi / 4)
        Point(x=-1.00, y=-1.00)
        """
        return cls(r * math.cos(theta), r * math.sin(theta))

    def __repr__(self):
        return f"{self.__class__.__name__}(x={self.x:.2f}, y={self.y:.2f})"

>>> Point.from_polar(r=-2**0.5, theta=math.pi / 4)
Point(x=-1.00, y=-1.00)

Conclusion

That's pretty much it! I hope you’ve learned something different and useful. Simple Factory methods are very cool and can abstract a lot of boilerplate. Not to mention that it makes your code clean and readable. In this post I showed how this pattern is used in the standard library and in other packages such as pandas.

Other posts you may like:

This post was originally published at https://miguendes.me/design-patterns-that-make-sense-in-python-simple-factory

See you next time!

3 Ways to Unit Test REST APIs in Python

Miguel Brito — Thu, 22 Jul 2021 08:08:25 +0000

In this tutorial, we’ll learn how to test REST API calls in Python.

In another words, we'll see the best ways to unit test code that performs HTTP requests by using mocks, design patterns, and the VCR library.

Unit tests are meant to test a single unit of behavior. In testing, a well-known rule of thumb is to isolate code that reaches external dependencies.

For instance, when testing a code that performs HTTP requests, it's recommended to replace the real call by a fake call during test time. This way we can to unit test it without performing a real HTTP request every time we run the test.

The question is, how can we isolate the code?

That’s exactly what you'll learn in this post! I’ll not only show you how to do it but also demonstrate the pros and cons of each approach.

📃 Table of Contents

✅ Requirements
🌦️ Demo App Using a Weather REST API
- Retrieving the Data
🧪 Testing the API Calls Using Mocks
🧪 Testing the API Using an Adapter
🧪 Testing the API Using VCR.py
😎 Conclusion

✅ Requirements

Python 3.8
pytest-mock
requests
flask
responses
VCR.py

🌦️ Demo App Using a Weather REST API

To put this problem in context, let's imagine that we're building a weather app. This app uses a third-party weather REST API to retrieve weather information for a particular city. One of the requirements is to generate a simple HTML page, like the image below.

To get the information about the weather, we must find it somewhere. Fortunately, OpenWeatherMap provides everything we need through its REST API service.

Ok, that's cool, but how can we use it?

We can get everything we need by sending a GET request to: 'https://api.openweathermap.org/data/2.5/weather?q={city_name}&appid={api_key}&units=metric'. For this tutorial, we’ll parametrize the city name and settle with the metric unit.

Retrieving the Data

To retrieve the weather data, we'll use requests. We can create a function that receives a city name as parameter and returns a json. The json will contain the temperature, weather description, sunset, sunrise time and so on.

The example below illustrates such function.

def find_weather_for(city: str) -> dict:
    """Queries the weather API and returns the weather data for a particular city."""
    url = API.format(city_name=city, api_key=API_KEY)
    resp = requests.get(url)
    return resp.json()

The URL is made up from two global variables.

BASE_URL = "https://api.openweathermap.org/data/2.5/weather"
API = BASE_URL + "?q={city_name}&appid={api_key}&units=metric"

The API returns a json in this format:

{
  "coord": {
    "lon": -0.13,
    "lat": 51.51
  },
  "weather": [
    {
      "id": 800,
      "main": "Clear",
      "description": "clear sky",
      "icon": "01d"
    }
  ],
  "base": "stations",
  "main": {
    "temp": 16.53,
    "feels_like": 15.52,
    "temp_min": 15,
    "temp_max": 17.78,
    "pressure": 1023,
    "humidity": 72
  },
  "visibility": 10000,
  "wind": {
    "speed": 2.1,
    "deg": 40
  },
  "clouds": {
    "all": 0
  },
  "dt": 1600420164,
  "sys": {
    "type": 1,
    "id": 1414,
    "country": "GB",
    "sunrise": 1600407646,
    "sunset": 1600452509
  },
  "timezone": 3600,
  "id": 2643743,
  "name": "London",
  "cod": 200
}

When we call resp.json(), it returns the data as a Python dictionary. In order to improve this data manipulation, we can represent them as a dataclass. This class has a factory method that gets the dictionary and returns a WeatherInfo instance.

This is good because we keep the representation stable. For example, if the API changes the way it structures the json, we can change the logic in just one place, in the from_dict method. So, other parts of the code won’t be affected. We can even get information from different sources and combining them in the from_dict method!

@dataclass
class WeatherInfo:
    temp: float
    sunset: str
    sunrise: str
    temp_min: float
    temp_max: float
    desc: str

    @classmethod
    def from_dict(cls, data: dict) -> "WeatherInfo":
        return cls(
            temp=data["main"]["temp"],
            temp_min=data["main"]["temp_min"],
            temp_max=data["main"]["temp_max"],
            desc=data["weather"][0]["main"],
            sunset=format_date(data["sys"]["sunset"]),
            sunrise=format_date(data["sys"]["sunrise"]),
        )

Now, let's create a function called retrieve_weather. We'll use this function to call the API and return a WeatherInfo so we can build our HTML page.

def retrieve_weather(city: str) -> WeatherInfo:
    """Finds the weather for a city and returns a WeatherInfo instance."""
    data = find_weather_for(city)
    return WeatherInfo.from_dict(data)

Good, we have the basic building blocks for our app. Before moving forward, let's unit test those functions.

🧪 1. Testing the API Using Mocks

According to wikipedia, a mock object is an object that simulates the behavior of a real object by mimicking it.

In Python, we can mock any object using the unittest.mock lib that is part of the standard library. To test the retrieve_weather function, we can then mock requests.get and return a static data.

`pytest-mock`

For this tutorial, we’ll use pytest as our testing framework of choice. pytest is a very extensible library that allows the extension through plugins.

To accomplish our mocking goals, we'll use pytest-mock. This plugin abstracts a bunch of setups from unittest.mock and makes our testing code very concise. If you are curious, I discuss more about it in another blog post.

Ok, enough talking, show me the code.

Here's a complete test case for the retrieve_weather function. This test uses two fixtures, one is the mocker fixture provided by the pytest-mock plugin. The other one is ours. It's just the static data we saved from a previous request.

@pytest.fixture()
def fake_weather_info():
    """Fixture that returns a static weather data."""
    with open("tests/resources/weather.json") as f:
        return json.load(f)

def test_retrieve_weather_using_mocks(mocker, fake_weather_info):
    """Given a city name, test that a HTML report about the weather is generated
    correctly."""
    # Creates a fake requests response object
    fake_resp = mocker.Mock()
    # Mock the json method to return the static weather data
    fake_resp.json = mocker.Mock(return_value=fake_weather_info)
    # Mock the status code
    fake_resp.status_code = HTTPStatus.OK

    mocker.patch("weather_app.requests.get", return_value=fake_resp)

    weather_info = retrieve_weather(city="London")
    assert weather_info == WeatherInfo.from_dict(fake_weather_info)

If we run the test, we get the following output:

============================= test session starts ==============================
...[omitted]...
tests/test_weather_app.py::test_retrieve_weather_using_mocks PASSED      [100%]
============================== 1 passed in 0.20s ===============================
Process finished with exit code 0

Great, our tests pass! But... Life is not a bed of roses. This test has pros and cons. Let's take a look at them.

Pros

Well, one pro we already discussed is that by mocking the return of the API we make our tests easier. We isolate the communication with the API and make the test predictable. It will always return what we want.

Cons

As cons, the problem is, what if we don’t want to use requests anymore and decide to go with the standard’s lib urllib. Every time we change the implementation of find_weather_for we will have to adapt the test. A good test is a test that doesn’t change when our implementation change. So, by mocking, we end up coupling our test with the implementation.

Also, another downside is the amount of setup we have to do before calling the function. At lest, 3 lines of code.

...
    # Creates a fake requests response object
    fake_resp = mocker.Mock()
    # Mock the json method to return the static weather data
    fake_resp.json = mocker.Mock(return_value=fake_weather_info)
    # Mock the status code
    fake_resp.status_code = HTTPStatus.OK
...

Can we do better?

Yes, please, follow along. Let's see now how to improve it a bit.

`responses`

Mocking requests using the mocker feature has the downside of having a long setup. A good way to avoid that is to use a library that intercepts requests calls and patch it. There are more than one lib for that, but simplest to me is responses. Let’s see how can we use it to replace mock.

@responses.activate
def test_retrieve_weather_using_responses(fake_weather_info):
    """Given a city name, test that a HTML report about the weather is generated
    correctly."""
    api_uri = API.format(city_name="London", api_key=API_KEY)
    responses.add(responses.GET, api_uri, json=fake_weather_info, status=HTTPStatus.OK)

    weather_info = retrieve_weather(city="London")
    assert weather_info == WeatherInfo.from_dict(fake_weather_info)

Again, this function makes use of our fake_weather_info fixture.

Good... let's run the test.

============================= test session starts ==============================
...
tests/test_weather_app.py::test_retrieve_weather_using_responses PASSED  [100%]
============================== 1 passed in 0.19s ===============================

Great! This test pass too. But... It's still not that great...

Pros

The good thing about using libraries like responses is that we don't need to patch requests ourselves. We save some setup by delegating the abstraction to the library. However, in case you haven't noticed, we have problems.

Cons

Again, the problem is, much like unittest.mock, our test is coupled to the implementation. If we replace requests, our test break.

🧪 2. Testing the API Using an Adapter

If by using mocks we couple our tests, what can we do?

Let’s image the following scenario: Say that we can no longer use requests and we’ll have to replace it by urllib, since it comes with Python. Not only that, we learned the lessons of not coupling test code with implementation and we want to avoid that in the future. We want to replace urllib and not have to rewrite the tests.

It turns out we can abstract away the code that performs the GET request.

Really? How?

We can abstract it by using an adapter. The adapter is a design pattern that is used to encapsulate, or wrap, the interface of other class and expose it as a new interface. This way we can change the adapters without changing our code. For example, we can encapsulate the details about requests in our find_weather_for and expose it via a function that takes only the URL.

So, this...

def find_weather_for(city: str) -> dict:
    """Queries the weather API and returns the weather data for a particular city."""
    url = API.format(city_name=city, api_key=API_KEY)
    resp = requests.get(url)
    return resp.json()

Becomes this:

def find_weather_for(city: str) -> dict:
    """Queries the weather API and returns the weather data for a particular city."""
    url = API.format(city_name=city, api_key=API_KEY)
    return adapter(url)

And the adapter becomes this:

def requests_adapter(url: str) -> dict:
    resp = requests.get(url)
    return resp.json()

Now it's time to refactor our retrieve_weather function:

def retrieve_weather(city: str) -> WeatherInfo:
    """Finds the weather for a city and returns a WeatherInfo instance."""
    data = find_weather_for(city, adapter=requests_adapter)
    return WeatherInfo.from_dict(data)

So, if we decide to change this implementation by an one that uses urllib, we can just swap the adapters.

def urllib_adapter(url: str) -> dict:
    """An adapter that encapsulates urllib.urlopen"""
    with urllib.request.urlopen(url) as response:
        resp = response.read()
    return json.loads(resp)

def retrieve_weather(city: str) -> WeatherInfo:
    """Finds the weather for a city and returns a WeatherInfo instance."""
    data = find_weather_for(city, adapter=urllib_adapter)
    return WeatherInfo.from_dict(data)

Ok, how about the tests?

To test retrieve_weather we can just create a fake adapter that is used during test time.

@responses.activate
def test_retrieve_weather_using_adapter(
    fake_weather_info,
):
    def fake_adapter(url: str):
        return fake_weather_info

    weather_info = retrieve_weather(city="London", adapter=fake_adapter)
    assert weather_info == WeatherInfo.from_dict(fake_weather_info)

If we run the test we get:

============================= test session starts ==============================
tests/test_weather_app.py::test_retrieve_weather_using_adapter PASSED    [100%]
============================== 1 passed in 0.22s ===============================

Pros

The pro for this approach is that we successfully decoupled our test from the implementation. We use dependency injection to inject a fake adapter during test time. Also, we can swap adapter at any time, including during runtime. We did all of this without changing the behavior.

Cons

The cons are that, since we’re using a fake adapter for tests, if we introduce a bug in the adapter we use in the implementation, our test won’t catch it. For example, let’s say that we pass a faulty parameter to requests, like this:

def requests_adapter(url: str) -> dict:
    resp = requests.get(url, headers=<some broken headers>)
    return resp.json()

This adapter will fail in production, and our unit tests won’t catch it. But truth to be told, we also have the same problem with the previous approach. That’s why we always need to go beyond unit tests and also have integration tests. But I’ll discuss them in another post. That being said, let’s consider another option.

🧪 3. Testing the API Using `VCR.py`

Now it’s finally the time to discuss our last option. I have only found about it quite recently, frankly. I’ve been using mocks for a long time and always had some problem with them. VCR.py is a library that simplifies a lot the tests that make HTTP requests.

It works by recording the HTTP interaction the first you run the test as a flat yaml file called cassette. Both the request and the response are serialized. When we run the test for the second time, VCR.py will intercept the call and return response for the request made.

Let's now see how to test retrieve_weather using VCR.py.

@vcr.use_cassette()
def test_retrieve_weather_using_vcr(fake_weather_info):
    weather_info = retrieve_weather(city="London")
    assert weather_info == WeatherInfo.from_dict(fake_weather_info)

Wow, is that it? No setup? What is that @vcr.use_cassette()?

Yes, that’s it! We have no setup, just a pytest annotation to tell VCR to intercept the call and save the cassette file.

How does the cassette file look like?

Good question, there's a bunch of things in it. This is because VCR saves every detail of the interaction.

interactions:
- request:
    body: null
    headers:
      Accept:
      - '*/*'
      Accept-Encoding:
      - gzip, deflate
      Connection:
      - keep-alive
      User-Agent:
      - python-requests/2.24.0
    method: GET
    uri: https://api.openweathermap.org/data/2.5/weather?q=London&appid=<YOUR API KEY HERE>&units=metric
  response:
    body:
      string: '{"coord":{"lon":-0.13,"lat":51.51},"weather":[{"id":800,"main":"Clear","description":"clearsky","icon":"01d"}],"base":"stations","main":{"temp":16.53,"feels_like":15.52,"temp_min":15,"temp_max":17.78,"pressure":1023,"humidity":72},"visibility":10000,"wind":{"speed":2.1,"deg":40},"clouds":{"all":0},"dt":1600420164,"sys":{"type":1,"id":1414,"country":"GB","sunrise":1600407646,"sunset":1600452509},"timezone":3600,"id":2643743,"name":"London","cod":200}'
    headers:
      Access-Control-Allow-Credentials:
      - 'true'
      Access-Control-Allow-Methods:
      - GET, POST
      Access-Control-Allow-Origin:
      - '*'
      Connection:
      - keep-alive
      Content-Length:
      - '454'
      Content-Type:
      - application/json; charset=utf-8
      Date:
      - Fri, 18 Sep 2020 10:53:25 GMT
      Server:
      - openresty
      X-Cache-Key:
      - /data/2.5/weather?q=london&units=metric
    status:
      code: 200
      message: OK
version: 1

That's a lot!

Indeed! The good thing is that we don’t need to care much about it. VCR.py takes care of that for us.

Pros

Now, for the pros, I can list at least five things:

No setup code
Tests remain isolated, so it’s fast
Tests are deterministic
If we change the request, like by using faulty headers, the test will fail.
Not coupled to the implementation, we can swap the adapters and the test will pass. The only thing that matters is that our request is the same.

Cons

Again, despite the enormous benefits compared to mocking, we still have problems.

If, for some reason, the API provider changes the format of the data, our test will still pass. Fortunately, this is not very frequent and API providers usually version their APIs before introducing such breaking changes. Also, unit tests are not meant to access the external API, so there isn’t much we can do here.

Another thing to consider is having end-to-end tests in place. These tests will call the server every time it runs. As the name says, it’s a more broad test and slow. They are meant to cover a lot more ground than unit tests. In fact, not every project will need to have them. So, in my view, VCR.py is more than enough for our needs.

😎 Conclusion

This is it, I hope you’ve learned something useful today. Testing API client applications can be a bit daunting. Yet, when armed with the right tools and knowledge, we can tame the beast.

You can find the full app on my github.

Other posts you may like:

See you next time!

This post was originally published at https://miguendes.me

The Best Way to Compare Two Dictionaries in Python

Miguel Brito — Sat, 17 Jul 2021 10:00:15 +0000

When I had to compare two dictionaries for the first time, I struggled―a lot!

For simple dictionaries, comparing them is usually straightforward. You can use the == operator, and it will work.

However, when you have specific needs, things become harder. The reason is, Python has no built-in feature allowing us to:

compare two dictionaries and check how many pairs are equal
assert nested dictionaries are equal (deep equality comparison)
find the difference between two dicts (dict diff)
compare dicts that have floating-point numbers as values

In this article, I will show how you can do those operations and many more, so let’s go.

Why You Need a Robust Way to Compare Dictionaries

Let's imagine the following scenario: you have two simple dictionaries. How can we assert if they match? Easy, right?

Yeah! You could use the == operator, off course!

>>> a = {
    'number': 1,
    'list': ['one', 'two']
}
>>> b = {
    'list': ['one', 'two'],
    'number': 1
}
>>> a == b
True

That's kind of expected, the dictionaries are the same. But what if some value is different, the result will be False but can we tell where do they differ?

>>> a = {
    'number': 1,
    'list': ['one', 'two']
}
>>> b = {
    'list': ['one', 'two'],
    'number': 2
}
>>> a == b
False

Hum... Just False doesn't tell us much...

What about the str's inside the list. Let's say that we want to ignore their cases.

>>> a = {
    'number': 1,
    'list': ['ONE', 'two']
}
>>> b = {
    'list': ['one', 'two'],
    'number': 1
}
>>> a == b
False

Oops...

What if the number was a float and we consider two floats to be the same if they have at least 3 significant digits equal? Put another way, we want to check if only 3 digits after the decimal point match.

>>> a = {
    'number': 1,
    'list': ['one', 'two']
}
>>> b = {
    'list': ['one', 'two'],
    'number': 1.00001
}
>>> a == b
False

You might also want to exclude some fields from the comparison. As an example, we might now want to remove the list key->value from the check. Unless we create a new dictionary without it, there's no method to do that for you.

Can't it get any worse?

Yes, what if a value is a numpy array?

>>> a = {
    'number': 1,
    'list': ['one', 'two'],
     'array': np.ones(3)
}
>>> b = {
    'list': ['one', 'two'],
    'number': 1,
    'array': np.ones(3)
}
>>> a == b
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-4-eeadcaeab874> in <module>
----> 1 a == b

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

Oh no, it raises an exception in the middle of our faces!

Damn it, what can we do then?

Using the Right Tool for the Job

Since dicts cannot perform advanced comparisons, there are only two forms of achieving that. You can either implement the functionality yourself or use a third party library. At some point in your life you probably heard about not reinventing the wheel. So that's precisely what we're going to do in this tutorial.

We'll adopt a library called deepdiff, from zepworks. deepdiff can pick up the difference between dictionaries, iterables, strings and other objects. It accomplishes that by searching for changes in a recursively manner.

deepdiff is not the only kid on the block, there's also Dictdiffer, developed by the folks at CERN. Dictdiffer is also cool but lacks a lot of the features that make deepdiff so interesting. In any case, I encourage you to look at both and determine which one works best for you.

This library is so cool that it not only works with dictionaries, but other iterables, strings and even custom objects. For example, you can "even mix and match" and take the difference between two lists of dicts.

Getting a Simple Difference

In this example, we'll be solving the first example I showed you. We want to find the key whose value differs between the two dicts. Consider the following code snippet, but this time using deepdiff.

In [1]: from deepdiff import DeepDiff

In [2]: a = {
   ...:     'number': 1,
   ...:     'list': ['one', 'two']
   ...: }

In [3]: b = {
   ...:     'list': ['one', 'two'],
   ...:     'number': 2
   ...: }

In [4]: diff = DeepDiff(a, b)

In [5]: diff
Out[5]: {'values_changed': {"root['number']": {'new_value': 2, 'old_value': 1}}}

Awesome! It tells us that the key 'number' had value 1 but the new dict, b, has a new value, 2.

Ignoring String Case

In our second example, we saw an example where one element of the list was in uppercase, but we didn't care about that. We wanted to ignore it and treat "one" as "ONE"

You can solve that by setting ignore_string_case=True

In [10]: a = {
    ...:     'number': 1,
    ...:     'list': ['ONE', 'two']
    ...: }
    ...: 

In [11]: b = {
    ...:     'list': ['one', 'two'],
    ...:     'number': 1
    ...: }

In [12]: diff = DeepDiff(a, b, ignore_string_case=True)

In [13]: diff
Out[13]: {}

If we don't do that, a very helpful message is printed.

In [14]: diff = DeepDiff(a, b)

In [15]: diff
Out[15]: 
{'values_changed': {"root['list'][0]": {'new_value': 'one',
   'old_value': 'ONE'}}}

Comparing Float Values

We also saw a case where we had a float number that we only wanted to check if the first 3 significant digits were equal. With DeepDiff it's possible to pass the exact number of digits AFTER the decimal point. Also, since floats differ from int's, we might want to ignore type comparison as well. We can solve that by setting ignore_numeric_type_changes=True.

In [16]: a = {
    ...:     'number': 1,
    ...:     'list': ['one', 'two']
    ...: }

In [17]: b = {
    ...:     'list': ['one', 'two'],
    ...:     'number': 1.00001
    ...: }

In [18]: diff = DeepDiff(a, b)

In [19]: diff
Out[19]: 
{'type_changes': {"root['number']": {'old_type': int,
   'new_type': float,
   'old_value': 1,
   'new_value': 1.00001}}}
In [24]: diff = DeepDiff(a, b, significant_digits=3, ignore_numeric_type_changes=True)

In [25]: diff
Out[25]: {}

Comparing `numpy` Values

When we tried comparing two dictionaries with a numpy array in it we failed miserably. Fortunately, DeepDiff has our backs here. It supports numpy objects by default!

In [27]: import numpy as np

In [28]: a = {
    ...:     'number': 1,
    ...:     'list': ['one', 'two'],
    ...:      'array': np.ones(3)
    ...: }

In [29]: b = {
    ...:     'list': ['one', 'two'],
    ...:     'number': 1,
    ...:     'array': np.ones(3)
    ...: }

In [30]: diff = DeepDiff(a, b)

In [31]: diff
Out[31]: {}

What if the arrays are different?

No problem!

In [28]: a = {
    ...:     'number': 1,
    ...:     'list': ['one', 'two'],
    ...:      'array': np.ones(3)
    ...: }

In [32]: b = {
    ...:     'list': ['one', 'two'],
    ...:     'number': 1,
    ...:     'array': np.array([1, 2, 3])
    ...: }

In [33]: diff = DeepDiff(a, b)

In [34]: diff
Out[34]: 
{'type_changes': {"root['array']": {'old_type': numpy.float64,
   'new_type': numpy.int64,
   'old_value': array([1., 1., 1.]),
   'new_value': array([1, 2, 3])}}}

It shows that not only the values are different but also the types!

Comparing Dictionaries With `datetime` Objects

Another common use case is comparing datetime objects. This kind of object has the following signature:

class datetime.datetime(year, month, day, hour=0, minute=0, second=0, microsecond=0, tzinfo=None, *, fold=0)

In case we have a dict with datetime objects, DeepDiff allows us to compare only certain parts of it. For instance, if only care about year, month, and day, then we can truncate it.

In [1]: import datetime

In [2]: from deepdiff import DeepDiff

In [3]: a = {
            'list': ['one', 'two'],
            'number': 1,
             'date': datetime.datetime(2020, 6, 17, 22, 45, 34, 513371)
        }

In [4]: b = {
            'list': ['one', 'two'],
            'number': 1,
            'date': datetime.datetime(2020, 6, 17, 12, 12, 51, 115791)
        }

In [5]: diff = DeepDiff(a, b, truncate_datetime='day')

In [6]: diff
Out[7]: {}

Comparing String Values

We've looked at interesting examples so far, and it's a common use case to use dicts to store strings values. Having a better way of contrasting them can help us a lot! In this section I'm going to explain you another lovely feature, the str diff.

In [13]: from pprint import pprint

In [17]: b = {
    ...:     'number': 1,
    ...:     'text': 'hi,\n my awesome world!'
    ...: }

In [18]: a = {
    ...:     'number': 1,
    ...:     'text': 'hello, my\n dear\n world!'
    ...: }

In [20]: ddiff = DeepDiff(a, b, verbose_level=2)

In [21]: pprint(ddiff, indent=2)
{ 'values_changed': { "root['text']": { 'diff': '--- \n'
                                                '+++ \n'
                                                '@@ -1,3 +1,2 @@\n'
                                                '-hello, my\n'
                                                '- dear\n'
                                                '- world!\n'
                                                '+hi,\n'
                                                '+ my awesome world!',
                                        'new_value': 'hi,\n my awesome world!',
                                        'old_value': 'hello, my\n'
                                                     ' dear\n'
                                                     ' world!'}}}

That's nice! We can see the exact lines where the two strings differ.

Excluding Fields

In this last example, I'll show you yet another common use case, excluding a field. We might want to exclude one or more items from the comparison. For instance, using the previous example, we might want to leave out the text field.

In [17]: b = {
    ...:     'number': 1,
    ...:     'text': 'hi,\n my awesome world!'
    ...: }

In [18]: a = {
    ...:     'number': 1,
    ...:     'text': 'hello, my\n dear\n world!'
    ...: }

In [26]: ddiff = DeepDiff(a, b, verbose_level=2, exclude_paths=["root['text']"])
    ...: 

In [27]: ddiff
Out[27]: {}

If you want even more advanced exclusions, DeepDiff also allow you to pass a regex expression. Check this out: https://zepworks.com/deepdiff/current/exclude_paths.html#exclude-regex-paths.

Conclusion

That's it for today, folks! I really hope you've learned something new and useful. Comparing dict's is a common use case since they can used to store almost any kind of data. As a result, having a proper tool to easy this effort is indispensable. DeepDiff has many features and can do reasonably advanced comparisons. If you ever need to compare dict's go check it out.

Other posts you may like:

See you next time!

This post was originally published at https://miguendes.me

Python F-String: 73 Examples to Help You Master It

Miguel Brito — Sat, 14 Nov 2020 11:38:05 +0000

Python f-strings are impressive!

Did you know you can use f-strings to string format almost anything in Python? You can use them to format floats, multiline strings, decimal places, objects and even use if-else conditionals within them.

In this post, I’ll show you at least 73 examples on how to format strings using Python 3's f-strings. You'll see the many ways you can take advantage of this powerful feature.

By the end of this guide, you'll have mastered:

how to use f string to format float numbers
how to format multiline string
how to define decimal places in a f-string
how to fix invalid syntax errors such as "syntaxerror: f-string: unmatched '['" or f-string: unmatched '('
how to use if else statement in a f-string
basic string interpolation formatting using f-strings
how to print f-strings
how to effectively add padding using fstring

Let's go!

What Are Python 3's F-Strings - a.k.a Literal String Interpolation?
How to Format Strings in Python 3 - The Basics
Limitations
How to Format an Expression
How to Use F-Strings to Debug Your Code
How to Format a Multiline F-String (Dealing with New Lines and Variables)
How to Fix F-String's Invalid Syntax Error
How to Fix "formatting a regular string which could be a f-string"
How to Format Numbers in Different Bases
How to Print Formatted Objects With F-Strings
How to Use F-Strings to Format a Float
How to Format a Number as Percentage
How to Justify or Add Padding to a F-String
How to Escape Characters With f-string
How to Add a Thousand Separator

15.1. How to Format a Number With Commas as Decimal Separator

15.2. How to Format a Number With Spaces as Decimal Separator
How to Format a Number in Scientific Notation (Exponential Notation)
Using if-else Conditional in a F-String
How to Use F-String With a Dictionary
How to Concatenate F-Strings
How to Format a Date With F-String
How to Add Leading Zeros
Conclusion

What Are Python F-Strings - a.k.a Literal String Interpolation?

String formatting has evolved quite a bit in the history of Python. Before Python 2.6, to format a string, one would either use the % operator, or string.Template module. Some time later, the str.format method came along and added to the language a more flexible and robust way of formatting a string.

Old string formatting with %:

>>> msg = 'hello world'
>>> 'msg: %s' % msg
'msg: hello world'

Using string.format:

>>> msg = 'hello world'
>>> 'msg: {}'.format(msg)
'msg: hello world'

To simplify formatting even further, in 2015, Eric Smith proposed the
PEP 498 -- Literal String Interpolation
, a new way to format a string for python 3.

PEP 498 presented this new string interpolation to be a simple and easy to use alternative to str.format. The only thing required is to put a 'f' before a string. And if you're new to the language, that's what f in Python means, it's a new syntax to create formatted strings.

Using f-strings:

>>> msg = 'hello world'
>>> f'msg: {msg}'
'msg: hello world'

And that was it! No need to use str.format or %. However, f-strings don’t replace str.format completely. In this guide I’ll show you an example where they are not suitable.

How to Format Strings in Python 3 - The Basics

As I have shown in the previous section, formatting strings in python using f-strings is quite straightforward. The sole requirement is to provide it a valid expression. f-strings can also start with capital F and you can combine with raw strings to produce a formatted output. However, you cannot mix them with bytes b"" or "u".

>>> book = "The dog guide"

>>> num_pages = 124

>>> f"The book {book} has {num_pages} pages"
'The book The dog guide has 124 pages'

>>> F"The book {book} has {num_pages} pages"
'The book The dog guide has 124 pages'

>>> print(Fr"The book {book} has {num_pages} pages\n")
The book The dog guide has 124 pages\n

>>> print(FR"The book {book} has {num_pages} pages\n")
The book The dog guide has 124 pages\n

>>> print(f"The book {book} has {num_pages} pages\n")
The book The dog guide has 124 pages

And that's pretty much it! In the next section, I'll show you several examples of everything you can do - and cannot do - with f-strings.

Limitations

Even though f-strings are very convenient, they don't replace str.format completely. f-strings evaluate expressions in the context where they appear. According the the PEP 498
, this means the expression has full access to local and global variables. They're also an expression evaluated at runtime. If the expression used inside the { <expr> } cannot be evaluated, the interpreter will raise an exception.

>>> f"{name}"
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-1-f0acc441190f> in <module>
----> 1 f"{name}"

NameError: name 'name' is not defined

This is not a problem for the str.format method, as you can define the template string and then call .format to pass on the context.

>>> s = "{name}"

>>> s.format(name="Python")
'Python'

>>> print(s)
{name}

Another limitation is that you cannot use inline comments inside a f-string.

>>> f"My name is {name #name}!"
  File "<ipython-input-37-0ae1738dd871>", line 1
    f"My name is {name #name}!"
    ^
SyntaxError: f-string expression part cannot include '#'

How to Format an Expression

If you don't want to define variables, you can use literals inside the brackets. Python will evaluate the expression and display the final result.

>>> f"4 * 4 is {4 * 4}"
'4 * 4 is 16'

Or if you prefer...

>>> n = 4

>>> f"4 * 4 is {n * n}"
'4 * 4 is 16'

How to Use F-Strings to Debug Your Code

One of most frequent usages of f-string is debugging. Before Python 3.8, many people would do hello = 42; f"hello = {hello}", but this is very repetitive. As a result, Python 3.8 brought a new feature. You can re-write that expression as f"{hello=}" and Python will display hello=42. The following example illustrates this using a function, but the principle is the same.

>>> def magic_number():
     ...:     return 42
     ...: 

>>> f"{magic_number() = }"
'magic_number() = 42'

How to Format a Multiline F-String (Dealing with New Lines and Variables)

You can use the newline character \n with f-strings to print a string in multiple lines.

>>> multi_line = (f'R: {color["R"]}\nG: {color["G"]}\nB: {color["B"
    ...: ]}\n')

>>> multi_line
'R: 123\nG: 145\nB: 255\n'

>>> print(multi_line)
R: 123
G: 145
B: 255

As an alternative, you can use triple quotes to represent the multiline string with variables. It not only allows you to add line breaks, it’s also possible to add TAB.

>>> other = f"""R: {color["R"]}
    ...: G: {color["G"]}
    ...: B: {color["B"]}
    ...: """

>>> print(other)
R: 123
G: 145
B: 255

Example with TABs.

>>> other = f'''
    ...: this is an example
    ...: 
    ...: ^Iof color {color["R"]}
    ...:     
    ...: '''

>>> other
'\nthis is an example\n\n\tof color 123\n    \n'

>>> print(other)

this is an example

    of color 123



>>>

How to Fix F-String's Invalid Syntax Error

If not used correctly, f-strings can raise a SyntaxError. The most common cause is using double-quotes inside a double quoted f-string. The same is also true for single quotes.

>>> color = {"R": 123, "G": 145, "B": 255}

>>> f"{color["R"]}"
  File "<ipython-input-43-1a7f5d512400>", line 1
    f"{color["R"]}"
    ^
SyntaxError: f-string: unmatched '['


# using only single quotes
>>> f'{color['R']}'
  File "<ipython-input-44-3499a4e3120c>", line 1
    f'{color['R']}'
    ^
SyntaxError: f-string: unmatched '['

This error not only happens with '[', but also '('. The cause is the same, it happens when you close a quote prematurely.

>>> print(f"price: {format(round(12.345), ",")}")
  File "<ipython-input-2-1ae6f786bc4d>", line 1
    print(f"price: {format(round(12.345), ",")}")
                                           ^
SyntaxError: f-string: unmatched '('

To fix that, you need to use single quotes.

>>> print(f"price: {format(round(12.345), ',')}")
price: 12

>>> color = {"R": 123, "G": 145, "B": 255}

>>> f"{color['R']}"
'123'

>>> f'{color["R"]}'
'123'

Another common case is to use f-strings in older versions of Python. f-strings were introduced in Python 3.6. If you use it in an older version, the interpreter will raise a SyntaxError: invalid syntax.

>>> f"this is an old version"
  File "<stdin>", line 1
    f"this is an old verion"
                            ^
SyntaxError: invalid syntax

If you see invalid syntax, make sure to double check the Python version you are running. In my case, I tested in on Python 2.7, and you can find the version by calling sys.version.

>>> import sys; print(sys.version)
2.7.18 (default, Apr 20 2020, 19:27:10) 
[GCC 8.3.0]

How to Fix "formatting a regular string which could be a f-string"

This error happens because pylint detects the old way of formatting string such as using % or the str.format method.

C0209: Formatting a regular string which could be a f-string (consider-using-f-string)

To fix that you can either:

replace the old formatting method with a f-string
ignore the pylint error using

In this post, I explain how to fix this issue step-by-step.

Replacing with f-strings

The following examples illustrate how to convert old method to f-string.

>>>  ip_address = "127.0.0.1"

# pylint complains if we use the methods below
>>> "http://%s:8000/" % ip_address
'http://127.0.0.1:8000/'

>>> "http://{}:8000/".format(ip_address)
'http://127.0.0.1:8000/'

# Replace it with a f-string
>>> f"http://{ip_address}:8000/"
'http://127.0.0.1:8000/'

Disable `pylint`

Alternatively, you can disable pylint by specifying the "disable" flag with the error code.

>>>  ip_address = "127.0.0.1"

# pylint complains if we use the methods below, so we can disable them
>>> "http://%s:8000/" % ip_address  # pylint: disable=C0209
'http://127.0.0.1:8000/'

>>> "http://{}:8000/".format(ip_address)  # pylint: disable=C0209
'http://127.0.0.1:8000/'

Another way of disabling that error is to add it to the .pylintrc file.

# .pylintrc
disable=
    ...
    consider-using-f-string,
    ...

There's yet another way of disabling it, which is by placing a comment at the top of the file, like so:

 # pylint: disable=consider-using-f-string

def your_function(fun):
    """Your code below"""
    ...

How to Format Numbers in Different Bases

f-strings also allow you to display an integer in different bases. For example, you can display an int as binary without converting it by using the b option.

>>> f'{7:b}'
'111'

In summary, you can use f-strings to format:

int to binary
int to hex
int to octal
int to HEX (where all chars are capitalized)

The following example uses the padding feature and the base formatting to create a table that displays an int in other bases.

>>> bases = {
       "b": "bin", 
       "o": "oct", 
       "x": "hex", 
       "X": "HEX", 
       "d": "decimal"
}
>>> for n in range(1, 21):
     ...:     for base, desc in bases.items():
     ...:         print(f"{n:5{base}}", end=' ')
     ...:     print()

    1     1     1     1     1 
   10     2     2     2     2 
   11     3     3     3     3 
  100     4     4     4     4 
  101     5     5     5     5 
  110     6     6     6     6 
  111     7     7     7     7 
 1000    10     8     8     8 
 1001    11     9     9     9 
 1010    12     a     A    10 
 1011    13     b     B    11 
 1100    14     c     C    12 
 1101    15     d     D    13 
 1110    16     e     E    14 
 1111    17     f     F    15 
10000    20    10    10    16 
10001    21    11    11    17 
10010    22    12    12    18 
10011    23    13    13    19 
10100    24    14    14    20

How to Print Formatted Objects With F-Strings

You can print custom objects using f-strings. By default, when you pass an object instance to a f-string, it will display what the __str__ method returns. However, you can also use the explicit conversion flag to display the __repr__.

!r - converts the value to a string using repr().
!s - converts the value to a string using str().

>>> class Color:
    def __init__(self, r: float = 255, g: float = 255, b: float = 255):
        self.r = r
        self.g = g
        self.b = b

    def __str__(self) -> str:
        return "A RGB color"

    def __repr__(self) -> str:
        return f"Color(r={self.r}, g={self.g}, b={self.b})"

>>> c = Color(r=123, g=32, b=255)

# When no option is passed, the __str__ result is printed
>>> f"{c}"
'A RGB color'

# When `obj!r` is used, the __repr__ output is printed
>>> f"{c!r}"
'Color(r=123, g=32, b=255)'

# Same as the default
>>> f"{c!s}"
'A RGB color'

Python also allows us to control the formatting on a per-type basis through the __format__ method. The following example shows how you can do all of that.

>>> class Color:
    def __init__(self, r: float = 255, g: float = 255, b: float = 255):
        self.r = r
        self.g = g
        self.b = b

    def __str__(self) -> str:
        return "A RGB color"

    def __repr__(self) -> str:
        return f"Color(r={self.r}, g={self.g}, b={self.b})"

    def __format__(self, format_spec: str) -> str:
        if not format_spec or format_spec == "s":
            return str(self)

        if format_spec == "r":
            return repr(self)

        if format_spec == "v":
            return f"Color(r={self.r}, g={self.g}, b={self.b}) - A nice RGB thing."

        if format_spec == "vv":
            return (
                f"Color(r={self.r}, g={self.g}, b={self.b}) "
                f"- A more verbose nice RGB thing."
            )

        if format_spec == "vvv":
            return (
                f"Color(r={self.r}, g={self.g}, b={self.b}) "
                f"- A SUPER verbose nice RGB thing."
            )

        raise ValueError(
            f"Unknown format code '{format_spec}' " "for object of type 'Color'"
        )

>>> c = Color(r=123, g=32, b=255)

>>> f'{c:v}'
'Color(r=123, g=32, b=255) - A nice RGB thing.'

>>> f'{c:vv}'
'Color(r=123, g=32, b=255) - A more verbose nice RGB thing.'

>>> f'{c:vvv}'
'Color(r=123, g=32, b=255) - A SUPER verbose nice RGB thing.'

>>> f'{c}'
'A RGB color'

>>> f'{c:s}'
'A RGB color'

>>> f'{c:r}'
'Color(r=123, g=32, b=255)'

>>> f'{c:j}'
--------------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-20-1c0ee8dd74be> in <module>
---------> 1 f'{c:j}'

<ipython-input-15-985c4992e957> in __format__(self, format_spec)
     29                 f"- A SUPER verbose nice RGB thing."
     30             )
--------> 31         raise ValueError(
     32             f"Unknown format code '{format_spec}' " "for object of type 'Color'"
     33         )

ValueError: Unknown format code 'j' for object of type 'Color'

Lastly, there's also the a option that escapes non-ASCII chars. For more info: https://docs.python.org/3/library/functions.html#ascii

>>> utf_str = "Áeiöu"

>>> f"{utf_str!a}"
"'\\xc1ei\\xf6u'"

How to Use F-Strings to Format a Float

f-strings allow format float numbers similar to str.format method. To do that, you can add a : (colon) followed by a . (dot) and the number of decimal places with a f suffix.

For instance, you can round a float to 2 decimal places and print the variable just like this:

>>> num = 4.123956

>>> f"num rounded to 2 decimal places = {num:.2f}"
'num rounded to 2 decimal places = 4.12'

If you don't specify anything, the float variable will use the full precision.

>>> print(f'{num}')
4.123956

How to Format a Number as Percentage

Python f-strings have a very convenient way of formatting percentage. The rules are similar to float formatting, except that you append a % instead of f. It multiplies the number by 100 displaying it in a fixed format, followed by a percent sign. You can also specify the precision.

>>> total = 87

>>> true_pos = 34

>>> perc = true_pos / total

>>> perc
0.39080459770114945

>>> f"Percentage of true positive: {perc:%}"
'Percentage of true positive: 39.080460%'

>>> f"Percentage of true positive: {perc:.2%}"
'Percentage of true positive: 39.08%'

How to Justify or Add Padding to a F-String

You can justify a string quite easily using < or > characters.

>>> greetings = "hello"

>>> f"She says {greetings:>10}"
'She says      hello'

# Pad 10 char to the right
>>> f"{greetings:>10}"
'     hello'

>>> f"{greetings:<10}"
'hello     '

# You can omit the < for left padding
>>> f"{greetings:10}"
'hello     '

>>> a = "1"

>>> b = "21"

>>> c = "321"

>>> d = "4321"

>>> print("\n".join((f"{a:>10}", f"{b:>10}", f"{c:>10}", f"{d:>10}")))
         1
        21
       321
      4321

How to Escape Characters With f-string

In case you want to display the variable name surrounded by the curly brackets instead of rendering its value, you can escape it using double {{<expr>}}.

>>> hello = "world"

>>> f"{{hello}} = {hello}"
'{hello} = world'

Now, if you want to escape a double quote, you can use the backslash \".

>>> f"{hello} = \"hello\""
'world = "hello"'

How to Center a String

Centering a string can be achieved by using var:^N where var is a variable you want to display and N is the string length. If N is shorter than the var, then Python display the whole string.

>>> hello = "world"

>>> f"{hello:^11}"
'   world   '

>>> f"{hello:*^11}"
'***world***'

# Extra padding is added to the right
>>> f"{hello:*^10}"
'**world***'

# N shorter than len(hello)
>>> f"{hello:^2}"
'world'

How To Add a Thousand Separator

f-strings also allow us to customize numbers. One common operation is to add an underscore to separate every thousand place.

>>> big_num = 1234567890

>>> f"{big_num:_}"
'1_234_567_890'

How to Format a Number With Commas as Decimal Separator

In fact, you can use any char as separator. It’s also possible to use a comma as separator.

>>> big_num = 1234567890

>>> f"{big_num:,}"
'1,234,567,890'

You can also format a float with commas and set the precision in one go.

>>> num = 2343552.6516251625

>>> f"{num:,.3f}"
'2,343,552.652'

How to Format a Number With Spaces as Decimal Separator

What about using spaces instead?

Well, this one is a bit “hacky” but it works. You can use the , as separator, then replace it with space.

>>> big_num = 1234567890

>>> f"{big_num:,}".replace(',', ' ')
'1 234 567 890'

Another option is to set the locale of your environment to one that uses spaces as a thousand separator such as pl_PL. For more info, see this thread on stack overflow.

How to Format a Number in Scientific Notation (Exponential Notation)

Formatting a number in scientific notation is possible with the e or E option.

>>> num = 2343552.6516251625

>>> f"{num:e}"
'2.343553e+06'

>>> f"{num:E}"
'2.343553E+06'

>>> f"{num:.2e}"
'2.34e+06'

>>> f"{num:.4E}"
'2.3436E+06'

Using `if-else` Conditional in a F-String

f-strings also evaluates more complex expressions such as inline if/else.

>>> a = "this is a"

>>> b = "this is b"

>>> f"{a if 10 > 5 else b}"
'this is a'

>>> f"{a if 10 < 5 else b}"
'this is b'

How to Use F-String With a Dictionary

You can use dictionaries in a f-string. The only requirement is to use a different quotation mark than the one enclosing the expression.

>>> color = {"R": 123, "G": 145, "B": 255}

>>> f"{color['R']}"
'123'

>>> f'{color["R"]}'
''123'

>>> f"RGB = ({color['R']}, {color['G']}, {color['B']})"
'RGB = (123, 145, 255)'

How to Concatenate F-Strings

Concatenating f-strings is like concatenating regular strings, you can do that implicitly, or explicitly by applying the + operator or using str.join method.

# Implicit string concatenation
>>> f"{123}" " = " f"{100}" " + " f"{20}" " + " f"{3}"
'123 = 100 + 20 + 3'

# Explicity concatenation using '+' operator
>>> f"{12}" + " != " + f"{13}"
'12 != 13'

# string concatenation using `str.join`
>>> " ".join((f"{13}", f"{45}"))
'13 45'

>>> "#".join((f"{13}", f"{45}"))
'13#45'

How to Format a Date With F-String

f-strings also support the formatting of datetime objects. The process is very similar to how str.format formats dates. For more info about the supported formats, check this table in the official docs.

>>> import datetime

>>> now = datetime.datetime.now()

>>> ten_days_ago = now - datetime.timedelta(days=10)

>>> f'{ten_days_ago:%Y-%m-%d %H:%M:%S}'
'2020-10-13 20:24:17'

>>> f'{now:%Y-%m-%d %H:%M:%S}'
'2020-10-23 20:24:17'

How to Add Leading Zeros

You can add leading zeros by adding using the format {expr:0len} where len is the length of the returned string. You can include a sign option. In this instance, + means the sign should be used for positive and negative numbers. The - is used only for negative numbers, which is the default behavior. For more info, check the string format specification page .

>>> num = 42

>>> f"{num:05}"
'00042'

>>> f'{num:+010}'
'+000000042'

>>> f'{num:-010}'
'0000000042'

>>> f"{num:010}"
'0000000042'

>>> num = -42

>>> f'{num:+010}'
'-000000042'

>>> f'{num:010}'
'-000000042'

>>> f'{num:-010}'
'-000000042'

Conclusion

That’s it for today, folks! I hope you’ve learned something different and useful. Knowing how to make the most out of f-string can make our lives so much easier. In this post, I showed the most common tricks I use in a day-to-day basis.

Other posts you may like:

See you next time!

7 pytest Features and Plugins That Will Save You Tons of Time

Miguel Brito — Sat, 10 Oct 2020 10:10:20 +0000

In this tutorial, we'll learn the best pytest features and plugins to speed up your development process. They're very simple and you can start using them right away.

How to Stop a Test Session on the First Failure
How to Re-Run Only the Last Failed Tests
How to Re-Run the Whole Test Session Starting With Last Failed Tests First
How to Display the Local Variables of a Failed Test
How to Run Only a Subset of Tests
How to Run Tests in Parallel
How to Re-Run Flaky Tests and Eliminate Intermittent Failures
Conclusion

How to Stop a Test Session on the First Failure

Running the full test suite of a big project may take a long time. No matter if you’re executing them locally or on a CI server, it’s always frustrating to see a test failing after waiting patiently. On certain occasions, you might wish to abort the entire session after the first failure so you can fix the broken test immediately. Fortunately, pytest comes with a very convenient CLI option for that, the -x / --exitfirst.

$ pytest -x tests/

How to Re-Run Only the Last Failed Tests

When developing locally, you might prefer to run all the tests before pushing to your repo. If you’re working on a small project with just a few tests, it’s OK to re-run all of them. However, if the full test suite takes minutes to run, you’ll probably want to execute only the ones that failed. pytest allows that via the --lf or --last-failed option. This way you can save precious time and iterate much quicker!

$ pytest --lf tests/

How to Re-Run the Whole Test Session Starting With Last Failed Tests First

Analogous to the preceding command, it might be helpful to re-run everything. The only difference is that you want to start with the failed test first. And to accomplish that, use the --ff or --failed-first flag.

$ pytest --ff tests/

How to Display the Local Variables of a Failed Test

We’ve learned how essential it is to iterate faster and how it can save you precious time. In the same fashion, it’s very crucial that we can pick up key hints that will help us debug the failed tests. By using the --showlocalsflag, or simply -l, we can see the value of all local variables in tracebacks.

$ pytest tests/test_variables.py -l               
================ test session starts ================
...                                                                                                                                                
tests/test_variables.py FF                                                                                                                                                     [100%]

================ FAILURES ================
_____________________________________________________________________________ test_local_variables[name] _____________________________________________________________________________

key = 'name'

    @pytest.mark.parametrize("key", ["name", "age"])
    def test_local_variables(key):
        result = person_info()
>       assert key in result
E       AssertionError: assert 'name' in {'height': 180}

key        = 'name'
result     = {'height': 180}

tests/test_variables.py:11: AssertionError
_____________________________________________________________________________ test_local_variables[age] ______________________________________________________________________________

key = 'age'

    @pytest.mark.parametrize("key", ["name", "age"])
    def test_local_variables(key):
        result = person_info()
>       assert key in result
E       AssertionError: assert 'age' in {'height': 180}

key        = 'age'
result     = {'height': 180}

tests/test_variables.py:11: AssertionError
================ short test summary info ================
FAILED tests/test_variables.py::test_local_variables[name] - AssertionError: assert 'name' in {'height': 180}
FAILED tests/test_variables.py::test_local_variables[age] - AssertionError: assert 'age' in {'height': 180}
================ 2 failed in 0.05s ================

How to Run Only a Subset of Tests

Sometimes you need to run just a subset of tests. One way of doing that is running all the test cases of an individual file. For example, you could do pytest test_functions.py. Although this is better than running everything, we can still improve it. By using the -k option, one can specify keyword expressions that pytest will use to select the tests to be executed.

# tests/test_variables.py
def test_asdict():
    ...

def test_astuple():
    ...

def test_aslist():
    ...

Say that you need to run the first two tests, you can pass a list of keywords separated by or:

$ pytest -k "asdict or astuple" tests/test_variables.py

Output:

$ pytest -k "asdict or astuple" tests/test_variables.py
==================================== test session starts ====================================
...                                          

tests/test_variables.py ..                                                            [100%]

============================== 2 passed, 1 deselected in 0.02s ==============================

How to Run Tests in Parallel

The more tests a project has, the longer it takes to run all of them. This sounds an indisputable statement, but it’s commonly overlooked. Running an extensive test suite one test after the other is an incredible waste of time. The best way to speed up the execution is by parallelizing it and taking advantage of multiple CPUs.

Sadly, pytest doesn’t have similar feature, so we must fall back on plugins. The best pytest plugin for that is pytest-xdist.

To send your tests to multiple CPUs, use the n or --numprocesses option.

$ pytest -n NUMCPUS

If you don’t know how many CPUs you have available, you can tell pytest-xdist to run the tests on all available CPUs with the auto value.

$ pytest -n auto

How to Re-Run Flaky Tests and Eliminate Intermittent Failures

One of the most disheartening situations is to see all tests passing locally only to fail on the CI server. Several reasons can cause failures like these, but mostly they’re a result of a “flaky” test. A “flaky” test is one that fails intermittently, in a non-deterministic manner. Usually, re-running them is enough to make them pass. The problem is, if you have a long running test suite, you need to re-trigger the CI step and wait several minutes. This is a great time sink and fortunately can be avoided.

So, to improve that, it is ideal that we can re-rerun the “flaky” test. By doing so, we can increase the chance of making it pass, avoiding the complete failure of the CI step.

The best pytest plugin for that is the pytest-rerunfailures. This plugin re-runs tests as much as we want, thus eliminating intermittent failures.

The simplest way to use it is to pass the --reruns option with the maximum number of times you’d like the tests to run.

$ pytest --reruns 5

If you know ahead of time that an individual test is flaky, you can just mark them to be re-run.

@pytest.mark.flaky(reruns=5)
def test_flaky():
    assert get_resut() is True

Conclusion

A large test suite can bring a lot of assurance to a project but it also comes with a cost. Long running testing sessions can be eat up a lot of development time and make iterations slower. By leveraging pytest features, and its plugin ecosystems, it’s possible to speed up the development process dramatically. In this tutorial, we looked at 7 tips we can adopt to improve our lives and waste less time executing tests.

If you liked this post, consider sharing it with your friends! Also, feel free to follow me https://miguendes.me.

7 pytest Features and Plugins That Will Save You Tons of Time first appeared on miguendes's blog.

Hacktoberfest: 69 Beginner-Friendly Projects You Can Contribute To

Miguel Brito — Tue, 29 Sep 2020 20:15:15 +0000

Wanna get started in open source and also score some nice swags? Join the Hacktoberfest® 2020!

In this post, I’ll list 69 beginner-friendly project that you can contribute to. By beginner-friendly I don’t mean simple. Some of them are very complex. However, like any other software project, there’s always some low hanging fruit.

The projects in this list have their issues triaged. That means the maintainers tag the simplest ones as “good first issue”, or "easy". Not only that, most of them ensure that only a novice must submit the pull request. That’s amazing, isn’t it?

By joining the Hacktoberfest, you can contribute with 4 pull requests during October 1 and October 31 and have a change to win a prize. The first 70,000 participants who conclude the challenge will be qualified to pick a limited edition T-shirt or plant a tree. However, be kind and only contribute if you have a value to offer back. Many people are only interested in the swags and that's kind of selfish in my opinion.

Without further ado, let’s check what we’ve got. I’ll break down the list by programming language. This way you can filter the projects by your favorite language.

I'm aware that I haven't covered all programming languages. However, you can also find a list of projects that tags #hacktoberfest here .

Disclaimer: I am not affiliated with Hacktoberfest®, or Digital Ocean in any way.

Javascript
Python
Java
TypeScript
Ruby
Rust
Go
PHP
C++
Kotlin

Javascript

https://github.com/responsively-org/responsively-app A modified web browser that helps in responsive web development. A web developer's must have dev-tool.
https://github.com/cypress-io/cypress
Fast, easy and reliable testing for anything that runs in a browser.
https://github.com/HabitRPG/habitica An open source habit building program which treats your life like a Role Playing Game.
https://github.com/vuejs/vue A progressive framework for building user interfaces.
https://github.com/nodejs/node An open-source, cross-platform, JavaScript runtime environment. It executes JavaScript code outside of a browser.
https://github.com/electron/electron A framework lets you write cross-platform desktop applications using JavaScript, HTML and CSS.
https://github.com/facebook/react-native A framework that brings React's declarative UI framework to iOS and Android.
https://github.com/gatsbyjs/gatsby A free and open source framework based on React that helps developers build blazing fast websites and apps
https://github.com/facebook/jest A delightful JavaScript Testing Framework with a focus on simplicity.
https://github.com/moment/moment A JavaScript date library for parsing, validating, manipulating, and formatting dates.
https://github.com/mochajs/mocha A simple, flexible, fun JavaScript test framework for Node.js & The Browser.
https://github.com/fastify/fastify Fast and low overhead web framework, for Node.js
https://github.com/vercel/next.js The React Framework
https://github.com/babel/babel A compiler for writing next generation JavaScript.

Python

https://github.com/scrapy/scrapy A fast high-level web crawling & scraping framework for Python.
https://github.com/mitmproxy/mitmproxy An interactive TLS-capable intercepting HTTP proxy for penetration testers and software developers.
https://github.com/pytest-dev/pytest A framework makes it easy to write small tests, yet scales to support complex functional testing
https://github.com/zulip/zulip A powerful, open source group chat application that combines the immediacy of real-time chat with the productivity benefits of threaded conversations.
https://github.com/ansible/ansible A radically simple IT automation platform that makes your applications and systems easier to deploy and maintain.
https://github.com/jupyter/notebook Jupyter Interactive Notebook
https://github.com/saltstack/salt Software to automate the management and configuration of any infrastructure or application at scale.
https://github.com/python/mypy Optional static typing for Python 3 and 2 (PEP 484)
https://github.com/opsdroid/opsdroid An open source chat-ops bot framework
https://github.com/pandas-dev/pandas Flexible and powerful data analysis / manipulation library for Python, providing labeled data structures similar to R data.frame objects, statistical functions, and much more
https://github.com/numpy/numpy The fundamental package for scientific computing with Python.
https://github.com/matplotlib/matplotlib A comprehensive library for creating static, animated, and interactive visualizations in Python.
https://github.com/arviz-devs/arviz Exploratory analysis of Bayesian models with Python

Java

https://github.com/strongbox/strongbox A modern OSS artifact repository manager.
https://github.com/elastic/elasticsearch Open Source, Distributed, RESTful Search Engine
https://github.com/commons-app/apps-android-commons The Wikimedia Commons Android app allows users to upload pictures from their Android phone/tablet to Wikimedia Commons
https://github.com/authorjapps/zerocode A community-developed, free, open source, API automation and load testing framework built using JUnit core runners for Http REST, SOAP, Security, Database, Kafka and much more.
https://github.com/JabRef/jabref Graphical Java application for managing BibTeX and biblatex (.bib) databases
https://github.com/sirixdb/sirix A temporal, evolutionary database system, which uses an accumulate only approach.
https://github.com/prestosql/presto A distributed SQL query engine for big data, under the auspices of the Presto Software Foundation.

TypeScript

https://github.com/microsoft/TypeScript A superset of JavaScript that compiles to clean JavaScript output.
https://github.com/microsoft/vscode Visual Studio Code
https://github.com/TroyTae/game-of-life Conway's game of life web version!

JRuby

https://github.com/jruby/jruby An implementation of Ruby on the JVM
https://github.com/sinatra/sinatra A classy web-development dressed in a DSL (official / canonical repo)
https://github.com/publiclab/plots2 A collaborative knowledge-exchange platform in Rails; we welcome first-time contributors! balloon
https://github.com/faker-ruby/faker A library for generating fake data such as names, addresses, and phone numbers.
https://github.com/chatwoot/chatwoot Open-source live chat software, an alternative to Intercom, Zendesk, Drift, Crisp etc.

Rust

https://github.com/servo/servo The Servo Browser Engine
https://github.com/rust-lang/rust-clippy A bunch of lints to catch common mistakes and improve your Rust code
https://github.com/rust-lang/rustfmt A tool for formatting Rust code according to style guidelines.
https://github.com/hyperium/hyper An HTTP library for Rust
https://github.com/nushell/nushell A new type of shell

Go

https://github.com/kubernetes/kubernetes Production-Grade Container Scheduling and Management
https://github.com/moby/moby A collaborative project for the container ecosystem to assemble container-based systems
https://github.com/hashicorp/terraform An open source tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.
https://github.com/mattermost/mattermost-server Open source Slack-alternative in Golang and React - Mattermost
https://github.com/gavv/httpexpect End-to-end HTTP and REST API testing for Go.
https://github.com/cockroachdb/cockroach An open source, cloud-native distributed SQL database.

PHP

https://github.com/phpmyadmin/phpmyadmin A web interface for MySQL and MariaDB
https://github.com/matomo-org/matomo The leading Free/Libre open analytics platform.
https://github.com/symfony/symfony An PHP web framework
https://github.com/nextcloud/server Nextcloud server, a safe home for all your data
https://github.com/PrestaShop/PrestaShop A fully scalable open source ecommerce solution.
https://github.com/appwrite/appwrite An end-to-end backend server for Web, Mobile, Native, or Backend apps packaged as a set of Docker microservices.
https://github.com/flarum/core Simple forum software for building great communities.

C++

https://github.com/godotengine/godot Multi-platform 2D and 3D game engine
https://github.com/projectM-visualizer/projectm A cross-platform music visualization. Open-source and Milkdrop-compatible Resources
https://github.com/roc-streaming/roc-toolkit Real-time audio streaming over the network.
https://github.com/amritansh22/Data-Structures-and-Algorithms-in-cpp This repository is in development phase and will soon provide you with c++ code of various data structures and algorithms
https://github.com/namanvats/cpalgorithms Algorithms and Techniques for competitive programming
https://github.com/noidawt/tomorrow A small graphical calculator project.

Kotlin

https://github.com/robstoll/atrium A multiplatform assertion library for Kotlin
https://github.com/hexagonkt/hexagon A microservices toolkit written in Kotlin.
https://github.com/sirixdb/sirix A temporal, evolutionary database system, which uses an accumulate only approach.

Conclusion

That's it folks! I hope you find a useful project to contribute during the Hacktoberfest and get started with the right foot in the open source world!

Other posts you may like:

If you liked this post, consider sharing it with your friends! Also, feel free to follow me https://miguendes.me.

Forem: Miguel Brito

15 Easy Ways to Trim a String in Python

How to Trim Characters From a String

Stripping Leading Whitespace From Beginning of a String

Stripping Trailing Whitespace From End of a String

Removing Spaces From From Start and End of a String

How to Trim Newlines

How to Trim Carriage Return (CRLF)

How to Trim Tabs

How to Trim a Combination of Characters From a String

How to Remove Multiple Spaces Inside a String

Removing Only Duplicates

Removing All Spaces

Using re (regex module)

Using replace

How to Strip a List of Strings

How to Strip an (Numpy) Array of Strings

Conclusion

How to Use datetime.timedelta in Python With Examples

Table of Contents

Adding Seconds, Minutes, Hours, Days, Weeks and Whatnot to a Date

How to Use timedelta to Add Days to a date or datetime Object

How to Use timedelta to Add Minutes to a datetime Object

How to Use timedelta to Add Weeks, Hours, Seconds, Milliseconds to a datetime

How to Add Years to a datetime in Python

How to Add Months to a datetime in Python

How to convert a timedelta to seconds, minutes, hours, or days

How to convert a timedelta to seconds

How to convert a timedelta to minutes

How to convert a timedelta to hours

How to convert a timedelta to days

How to convert a timedelta to years

How to Take the Difference Between Two Dates

How to Calculate the Number of Days Between Two Dates

How to Calculate the Number of Minutes Between Two Dates

How to Format a timedelta as string

Conclusion

7 Different Ways to Flatten a List of Lists in Python

Table of Contents

Flattening a list of lists with list comprehensions

How to flatten list of strings, tuples or mixed types

How to flatten a list and remove duplicates

Flattening a list of lists with the sum function

Flattening using itertools.chain

Flatten a regular list of lists with numpy

Flattening irregular lists

The recursive approach

The iterative approach

Which method is faster? A performance comparison

Flatten generator comprehension

Flatten sum

Flatten chain

Flatten numpy

Flatten recursive

Flatten deque

Conclusion

How to Check if an Exception Is Raised (or Not) With pytest

TL;DR

Introduction

Table of Contents

How to Assert That an Exception Is Raised

How to Assert the Exception Message - And Type

How to Assert That NO Exception Is Raised

Conclusion

How to Pass Multiple Arguments to a map Function in Python

Introduction

Table of Contents

The Problem With the map() Function

Solution 1 - Mapping Multiple Arguments with itertools.starmap()

Solution 2 - Using functools.partial to “Freeze” the Arguments

Solution 3 - Mapping Multiple Arguments by "Repeating" Them

Problem 2: Passing Multiple Parameters to multiprocessing Pool.map

Using pool.starmap

Using partial()

Problem 3: How to Pass Multiple Arguments to concurrent futures Executor.map?

Using partial() With a ProcessPoolExecutor (or ThreadPoolExecutor)

Using repeat()

Conclusion

What if Python Had This Ruby Feature?

Why?

Using `re` (regex module)

Using `replace`

How to Use `timedelta` to Add Days to a `date` or `datetime` Object

How to Use `timedelta` to Add Minutes to a `datetime` Object

How to Use `timedelta` to Add Weeks, Hours, Seconds, Milliseconds to a `datetime`

How to Add Years to a `datetime` in Python

How to Add Months to a `datetime` in Python

How to convert a `timedelta` to seconds, minutes, hours, or days

How to convert a `timedelta` to seconds

How to convert a `timedelta` to minutes

How to convert a `timedelta` to hours

How to convert a `timedelta` to days

How to convert a `timedelta` to years

How to Format a `timedelta` as string

Flattening a list of lists with the `sum` function

Flattening using `itertools.chain`

The Problem With the `map()` Function

Solution 1 - Mapping Multiple Arguments with `itertools.starmap()`

Solution 2 - Using `functools.partial` to “Freeze” the Arguments

Problem 2: Passing Multiple Parameters to multiprocessing `Pool.map`

Using `pool.starmap`

Using `partial()`

Problem 3: How to Pass Multiple Arguments to concurrent futures `Executor.map`?

Using `partial()` With a ProcessPoolExecutor (or ThreadPoolExecutor)

Using `repeat()`

`pytest-mock`

`responses`

🧪 3. Testing the API Using `VCR.py`

Comparing `numpy` Values

Comparing Dictionaries With `datetime` Objects

Disable `pylint`