Forem: Sundeep

I wrote a short, introductory book for Python

Sundeep — Fri, 12 Feb 2021 13:07:48 +0000

Hello!

I recently published my ebook titled "100 Page Python Intro". This book is a short, introductory guide for the Python programming language. This book is well suited:

As a reference material for Python beginner workshops
If you have prior experience with another programming language
If you want a complement resource after reading a Python basics book, watching a video course, etc

Ebook links

🎉🎉 To celebrate, I'm giving away FREE PDF/EPUB versions (until 17 Feb, 2021) along with some of my other books. You can get them from Gumroad/Leanpub using the links given below:

100 Page Python Intro
- https://gumroad.com/l/100pagepythonintro
- https://leanpub.com/100pagepythonintro
Python re(gex)?
- https://gumroad.com/l/py_regex
- https://leanpub.com/py_regex
Magical one-liners bundle
- https://gumroad.com/l/oneliners
- https://leanpub.com/b/oneliners

Web version

You can also read the entire book as a series of posts here:

Python Introduction

Sundeep ・ Jan 8 ・ 7 min read

#python #beginners #ebook

GitHub repo

The https://github.com/learnbyexample/100_page_python_intro repo has program/example files, markdown source and other details about the book.

Preface
Introduction
Numeric data types
Strings and user input
Defining functions
Control structures
Importing and creating modules
Installing modules and Virtual environments
Exception handling
Debugging
Testing
Tuple and Sequence operations
List
Mutability
Dict
Set
Text processing
Comprehensions and Generator expressions
Dealing with files
Executing external commands
Command line arguments

Motivation and FAQ

I've been conducting a few Python introduction workshops for college students and faculty for the past four years (which came to a premature end thanks to the pandemic). These students were already familiar with another programming languages such as C, Java, etc. I used to provide my notes in PDF format as a workshop reference material, further reading resources, etc. After I started writing a book titled Practice Python Projects, I realized I'd be better served by improving my Python knowledge first. What better way to do it than writing a book? And it did teach me a lot of things, some of the highlights being:

Exploring docs.python: Glossary
You cannot have mutable objects as a set element!
startswith() and endswith() string methods support tuple argument for testing multiple substrings
Command line options like -q, -B, etc

Why is it called "100 Page Python Intro" when it has more than 100 pages?

There are 2 hard problems in computer science: cache invalidation, naming things, and off-by-1 errors — Leon Bambrick

The material I was using for my workshops was 56 pages. I had more chapters to add, but I thought it would be a struggle to reach 100 pages, instead of overshooting the goal in the end. The measurement also depends on a few factors. The main content will be less than 100 pages if I reduce the font size from 12 to 11, exclude cover, TOC, Preface, etc.

Feedback

Hope you find it useful and fun to learn Python. As always, I'd highly appreciate your feedback. Please do let me know if you spot any error or typo.

Happy learning :)

CLI apps with Python

Sundeep — Wed, 03 Feb 2021 13:31:12 +0000

This chapter will show a few examples of processing CLI arguments using sys and argparse modules. The fileinput module is also introduced in this chapter, which is handy for in-place file editing.

sys.argv

Command line arguments passed when executing a Python program can be accessed as a list of strings via sys.argv. The first element (index 0) contains the name of the Python script or -c or empty string, depending upon how the Python interpreter was called. Rest of the elements will have the command line arguments, if any were passed along the script to be executed. See docs.python: sys.argv for more details.

Here's a program that accepts two numbers passed as CLI arguments and displays the sum only if the input was passed correctly.

# sum_two_nums.py
import ast
import sys

try:
    num1, num2 = sys.argv[1:]
    total = ast.literal_eval(num1) + ast.literal_eval(num2)
except ValueError:
    sys.exit('Error: Please provide exactly two numbers as arguments')
else:
    print(f'{num1} + {num2} = {total}')

The ast.literal_eval() method is handy for converting a string value to built-in literals, especially for collection data types. If you wanted to use int() and float() for the above program, you'd have to add logic for separating the input into integers and floating-point first. Passing a string to sys.exit() gets printed to the stderr stream and sets the exit status as 1 in addition to terminating the script.

Here's a sample run:

$ python3.9 sum_two_nums.py 2 3.14
2 + 3.14 = 5.140000000000001
$ echo $?
0

$ python3.9 sum_two_nums.py 2 3.14 7
Error: Please provide exactly two numbers as arguments
$ echo $?
1
$ python3.9 sum_two_nums.py 2 abc
Error: Please provide exactly two numbers as arguments

As an exercise, modify the above program to handle TypeError exceptions. Instead of the output shown below, inform the user about the error using sys.exit() method.

$ python3.9 sum_two_nums.py 2 [1]
Traceback (most recent call last):
  File "/home/learnbyexample/Python/programs/sum_two_nums.py", line 6, in <module>
    total = ast.literal_eval(num1) + ast.literal_eval(num2)
TypeError: unsupported operand type(s) for +: 'int' and 'list'

As another exercise, accept one or more numbers as input arguments. Calculate and display the following details about the input — sum, product and average.

In-place editing with fileinput

To edit a file in-place, the fileinput module comes in handy. Here's a program that loops over filenames passed as CLI arguments (i.e. sys.argv[1:]), does some processing and writes back the changes to the original input files. You can also provide one or more filenames to the files keyword argument, if you do not wish to pass them as CLI arguments.

# inplace_edit.py
import fileinput

with fileinput.input(inplace=True) as f:
    for ip_line in f:
        op_line = ip_line.rstrip('\n').capitalize() + '.'
        print(op_line)

Note that unlike open(), the FileInput object doesn't support write() method. However, using print() is enough. Here's a sample run:

$ python3.9 inplace_edit.py [io]p.txt

$ # check if files have changed
$ cat ip.txt
Hi there.
Today is sunny.
Have a nice day.
$ cat op.txt
This is a sample line of text.
Yet another line.

$ # if stdin is passed as input, inplace gets disabled
$ echo 'GooD moRNiNg' | python3.9 inplace_edit.py
Good morning.

As inplace=True permanently modifies your input files, it is always a good idea to check your logic on sample files first. That way your data wouldn't be lost because of an error in your program. You can also ask fileinput to create backups if you need to recover original files later — for example, backup='.bkp' will create backups by adding .bkp as the suffix to the original filenames.

argparse

sys.argv is good enough for simple use cases. If you wish to create a CLI application with various kinds of flags and arguments (some of which may be optional/mandatory) and so on, use a module such as the built-in argparse or a third-party solution like click.

Quoting from docs.python: argparse:

The argparse module makes it easy to write user-friendly command-line interfaces. The program defines what arguments it requires, and argparse will figure out how to parse those out of sys.argv. The argparse module also automatically generates help and usage messages and issues errors when users give the program invalid arguments.

Here's a CLI application that accepts a file containing a list of filenames that are to be sorted by their extension. Files with the same extension are further sorted in ascending order. The program also implements an optional flag to remove duplicate entries.

# sort_ext.py
import argparse

parser = argparse.ArgumentParser()
parser.add_argument('-f', '--file', required=True,
                    help="input file to be sorted")
parser.add_argument('-u', '--unique', action='store_true',
                    help="sort uniquely")
args = parser.parse_args()

ip_lines = open(args.file).readlines()
if args.unique:
    ip_lines = set(ip_lines)

op_lines = sorted(ip_lines, key=lambda s: (s.rsplit('.', 1)[-1], s))
for line in op_lines:
    print(line, end='')

The documentation for the CLI application is generated automatically based on the information passed to the parser. You can use help options (which is added automatically too) to view the documentation, as shown below:

$ python3.9 sort_ext.py -h
usage: sort_ext.py [-h] -f FILE [-u]

optional arguments:
  -h, --help            show this help message and exit
  -f FILE, --file FILE  input file to be sorted
  -u, --unique          sort uniquely

$ python3.9 sort_ext.py
usage: sort_ext.py [-h] -f FILE [-u]
sort_ext.py: error: the following arguments are required: -f/--file

The add_argument() method allows you to add details about an option/argument for the CLI application. The first parameter names an argument or option (starts with -). The help keyword argument lets you add documentation for that particular option/argument. See docs.python: add_argument for documentation and details about other keyword arguments.

The above program adds two options, one to store the filename to be sorted and the other to act as a flag for sorting uniquely. Here's a sample text file that needs to be sorted based on the extension.

$ cat sample.txt
input.log
basic.test
input.log
out.put.txt
sync.py
input.log
async.txt

Here's the output with both types of sorting supported by the program.

# default sort
$ python3.9 sort_ext.py -f sample.txt
input.log
input.log
input.log
sync.py
basic.test
async.txt
out.put.txt

# unique sort
$ python3.9 sort_ext.py -uf sample.txt
input.log
sync.py
basic.test
async.txt
out.put.txt

See docs.python HOWTOs: Argparse Tutorial for a more detailed introduction.

Accepting stdin

CLI tools like grep, sed, awk and many others can accept data from stdin as well as accept filenames as arguments. The previous program modified to add stdin functionality is shown below. args.file is now a positional argument instead of an option. nargs='?' indicates that this argument is optional. type=argparse.FileType('r') allows you to automatically get a filehandle in read mode for the filename supplied as an argument. If filename isn't provided, default=sys.stdin kicks in and you get a filehandle for the stdin data.

# sort_ext_stdin.py
import argparse, sys

parser = argparse.ArgumentParser()
parser.add_argument('file', nargs='?',
                    type=argparse.FileType('r'), default=sys.stdin,
                    help="input file to be sorted")
parser.add_argument('-u', '--unique', action='store_true',
                    help="sort uniquely")
args = parser.parse_args()

ip_lines = args.file.readlines()
if args.unique:
    ip_lines = set(ip_lines)

op_lines = sorted(ip_lines, key=lambda s: (s.rsplit('.', 1)[-1], s))
for line in op_lines:
    print(line, end='')

Here's the help for the modified program:

$ python3.9 sort_ext_stdin.py -h
usage: sort_ext_stdin.py [-h] [-u] [file]

positional arguments:
  file          input file to be sorted

optional arguments:
  -h, --help    show this help message and exit
  -u, --unique  sort uniquely

Here's a sample run showing both stdin and filename argument functionality.

# 'cat' is used here for illustration purposes only
$ cat sample.txt | python3.9 sort_ext_stdin.py
input.log
input.log
input.log
sync.py
basic.test
async.txt
out.put.txt
$ python3.9 sort_ext_stdin.py -u sample.txt
input.log
sync.py
basic.test
async.txt
out.put.txt

As an exercise, add -o, --output optional argument to store the output in a file for the above program.

Executing external commands from Python

Sundeep — Wed, 03 Feb 2021 13:29:14 +0000

This chapter will discuss how you can execute external commands from Python, capture their output and other relevant details such as the exit status. The availability of commands depends on the OS you are using (mine is Linux).

Using os module

Last chapter showed a few examples with os module for file processing. The os module is a feature rich module with lot of other uses, like providing an interface for working with external commands. Here's an example:

>>> import os

>>> os.system('echo hello "$USER"')
hello learnbyexample
0

Similar to the print() function, the output of the external command, if any, is displayed on the screen. The return value is the exit status of the command, which gets displayed by default on the REPL. 0 means the command executed successfully, any other value indicates some kind of failure. As per docs.python: os.system:

On Unix, the return value is the exit status of the process encoded in the format specified for wait().

Here's an example for non-zero exit status:

>>> status = os.system('ls xyz.txt')
ls: cannot access 'xyz.txt': No such file or directory
>>> status
512
# to get the actual exit value
>>> os.waitstatus_to_exitcode(status)
2

# if you don't want to see the error message,
# you can redirect the stderr stream
>>> os.system('ls xyz.txt 2> /dev/null')
512

You can use the os.popen() method to save the results of an external command. It provides a file object like interface for both read (default) and write. To check the status, call close() method on the filehandle (None means success).

>>> fh = os.popen('wc -w <ip.txt')
>>> op = fh.read()
>>> op
'9\n'
>>> status = fh.close()
>>> print(status)
None

# if you just want the output
>>> os.popen('wc -w <ip.txt').read()
'9\n'

subprocess.run

The subprocess module provides a more flexible and secure option to execute external commands, at the cost of being more verbose.

Quoting relevant parts from doc.python: subprocess module:

The subprocess module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes.

The recommended approach to invoking subprocesses is to use the run() function for all use cases it can handle. For more advanced use cases, the underlying Popen interface can be used directly.

>>> import subprocess

>>> subprocess.run('pwd')
'/home/learnbyexample/Python/programs/'
CompletedProcess(args='pwd', returncode=0)

>>> process = subprocess.run(('ls', 'xyz.txt'))
ls: cannot access 'xyz.txt': No such file or directory
>>> process.returncode
2

The first argument to run() method is the command to be executed. This can be either be a single string or a sequence of strings (if you need to pass arguments to the command being executed). By default, command output is displayed on the screen. Return value is a CompletedProcess object, which has relevant information for the command that was executed such as the exit status.

As an exercise, read subprocess.run documentation and modify the above ls example to:

redirect the stderr stream to /dev/null
automatically raise an exception when the exit status is non-zero

shell=True

You can also construct a single string command, similar to os.system(), if you set shell keyword argument to True. While this is convenient, use it only if you have total control over the command being executed such as your personal scripts. Otherwise, it can lead to security issues, see stackoverflow: why not use shell=True for details.

Quoting from docs.python: subprocess Frequently Used Arguments:

If shell is True, the specified command will be executed through the shell. This can be useful if you are using Python primarily for the enhanced control flow it offers over most system shells and still want convenient access to other shell features such as shell pipes, filename wildcards, environment variable expansion, and expansion of ~ to a user's home directory

>>> p = subprocess.run(('echo', '$HOME'))
$HOME
>>> p = subprocess.run('echo $HOME', shell=True)
/home/learnbyexample

>>> p = subprocess.run(('ls', '*.txt'))
ls: cannot access '*.txt': No such file or directory
>>> p = subprocess.run('ls *.txt', shell=True)
ip.txt

>>> p = subprocess.run('seq -s, 10 > out.txt', shell=True)
>>> p = subprocess.run('cat out.txt', shell=True)
1,2,3,4,5,6,7,8,9,10

If shell=True cannot be used but shell features as mentioned above is needed, you can use modules like os, glob, shutil and so on as applicable. See also docs.python: Replacing Older Functions with the subprocess Module.

>>> p = subprocess.run(('echo', os.getenv('HOME')))
/home/learnbyexample

Changing shell

By default, /bin/sh is the shell used for POSIX systems. You can change that by setting the executable argument to the shell of your choice.

>>> p = subprocess.run('diff <(seq 3) <(seq 4)', shell=True)
/bin/sh: 1: Syntax error: "(" unexpected

>>> p = subprocess.run('diff <(seq 3) <(seq 4)', shell=True,
                       executable='/bin/bash')
3a4
> 4

Capture output

If you use capture_output=True, the CompletedProcess object will provide stdout and stderr results as well. These are provided as bytes data type by default. You can change that by setting text=True.

>>> p = subprocess.run(('date', '-u', '+%A'), capture_output=True, text=True)
>>> p
CompletedProcess(args=('date', '-u', '+%A'), returncode=0,
                 stdout='Monday\n', stderr='')
>>> p.stdout
'Monday\n'

You can also use subprocess.check_output() method to directly get the output.

>>> subprocess.check_output(('date', '-u', '+%A'), text=True)
'Monday\n'

You can also use legacy methods subprocess.getstatusoutput() and subprocess.getoutput() but they lack in features and do not provide secure options. See docs.python: subprocess Legacy Shell Invocation Functions for details.

Dealing with files in Python

Sundeep — Tue, 02 Feb 2021 15:12:35 +0000

This chapter will discuss the open() built-in function and some of the built-in modules for file processing.

open and close

The open() built-in function is one of the ways to read and write files. The first argument to this function is the filename to be processed. The filename is a relative/absolute path to the location of the file. Rest are keyword arguments that you can configure. The output is a TextIOWrapper object (i.e. a filehandle), which you can use as an iterator. Here's an example:

# default mode is rt i.e. read text
>>> fh = open('ip.txt')
>>> fh
<_io.TextIOWrapper name='ip.txt' mode='r' encoding='UTF-8'>
>>> next(fh)
'hi there\n'
>>> next(fh)
'today is sunny\n'
>>> next(fh)
'have a nice day\n'
>>> next(fh)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

# check if the filehandle is active or closed
>>> fh.closed
False
# close the filehandle
>>> fh.close()
>>> fh.closed
True

The mode argument specifies what kind of processing you want. Only text mode will be covered in this chapter, which is the default. You can combine options, for example, rb means read in binary mode. Here's the relevant details from the documentation:

'r' open for reading (default)
'w' open for writing, truncating the file first
'x' open for exclusive creation, failing if the file already exists
'a' open for writing, appending to the end of the file if it exists
'b' binary mode
't' text mode (default)
'+' open for updating (reading and writing)

The encoding argument is meaningful only in the text mode. You can check the default encoding for your environment using the locale module as shown below. See docs.python: standard encodings and docs.python: Unicode for more details.

>>> import locale
>>> locale.getpreferredencoding()
'UTF-8'

Here's how Python handles line separation by default, see documentation for more details.

On input, if newline is None, universal newlines mode is enabled. Lines in the input can end in '\n', '\r', or '\r\n', and these are translated into '\n' before being returned to the caller.

On output, if newline is None, any '\n' characters written are translated to the system default line separator, os.linesep.

If the given filename doesn't exist, you'll get a FileNotFoundError exception.

>>> open('xyz.txt', 'r', encoding='ascii')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
FileNotFoundError: [Errno 2] No such file or directory: 'xyz.txt'

Context manager

Quoting from docs.python: Reading and Writing Files:

It is good practice to use the with keyword when dealing with file objects. The advantage is that the file is properly closed after its suite finishes, even if an exception is raised at some point. Using with is also much shorter than writing equivalent try-finally blocks.

# read_file.py
with open('ip.txt', 'r', encoding='ascii') as f:
    for ip_line in f:
        op_line = ip_line.rstrip('\n').capitalize() + '.'
        print(op_line)

Recall that as keyword was introduced in the try-except section. Here's the output of the above program:

$ python3.9 read_file.py
Hi there.
Today is sunny.
Have a nice day.

See The Magic of Python Context Managers for more details.

read, readline and readlines

The read() method gives you entire remaining contents of the file as a single string. The readline() method gives next line of text and readlines() gives all the remaining lines as a list of strings.

>>> open('ip.txt').read()
'hi there\ntoday is sunny\nhave a nice day\n'

>>> fh = open('ip.txt')
# 'readline' is similar to 'next'
# but returns empty string instead of StopIteration exception
>>> fh.readline()
'hi there\n'
>>> fh.readlines()
['today is sunny\n', 'have a nice day\n']
>>> fh.readline()
''

write

# write_file.py
with open('op.txt', 'w', encoding='ascii') as f:
    f.write('this is a sample line of text\n')
    f.write('yet another line\n')

You can call the write() method on a filehandle to add contents to that file (provided the mode you have set supports writing). Unlike print(), the write() method doesn't automatically add newline characters.

$ python3.9 write_file.py 

$ cat op.txt
this is a sample line of text
yet another line

$ file op.txt
op.txt: ASCII text

If the file already exists, the w mode will overwrite the contents (i.e. existing content will be lost).

You can also use the print() function for writing by passing the filehandle to the file argument. The fileinput module supports in-place editing and other features (examples will be discussed later).

File processing modules

This section gives introductory examples for some of the built-in modules that are handy for file processing. Quoting from docs.python: os:

This module provides a portable way of using operating system dependent functionality.

>>> import os

# current working directory
>>> os.getcwd()
'/home/learnbyexample/Python/programs/'

# value of an environment variable
>>> os.getenv('SHELL')
'/bin/bash'

# file size
>>> os.stat('ip.txt').st_size
40

# check if given path is a file
>>> os.path.isfile('ip.txt')
True

Quoting from docs.python: glob:

The glob module finds all the pathnames matching a specified pattern according to the rules used by the Unix shell, although results are returned in arbitrary order. No tilde expansion is done, but *, ?, and character ranges expressed with [] will be correctly matched.

>>> import glob

# list of files (including directories) containing '_file' in their name
>>> glob.glob('*_file*')
['read_file.py', 'write_file.py']

Quoting from docs.python: shutil:

The shutil module offers a number of high-level operations on files and collections of files. In particular, functions are provided which support file copying and removal.

>>> import shutil

>>> shutil.copy('ip.txt', 'ip_file.txt')
'ip_file.txt'
>>> glob.glob('*_file*')
['read_file.py', 'ip_file.txt', 'write_file.py']

Quoting from docs.python: pathlib:

This module offers classes representing filesystem paths with semantics appropriate for different operating systems. Path classes are divided between pure paths, which provide purely computational operations without I/O, and concrete paths, which inherit from pure paths but also provide I/O operations.

>>> from pathlib import Path

# use 'rglob' instead of 'glob' if you want to match names recursively
>>> list(Path('programs').glob('*file.py'))
[PosixPath('programs/read_file.py'), PosixPath('programs/write_file.py')]

See pathlib module: taming the file system and stackoverflow: How can I iterate over files in a given directory? for more details and examples.

There are specialized modules for structured data processing as well, for example:

Exercises

Write a program that reads a known filename f1.txt which contains a single column of numbers in Python syntax. Your task is to display the sum of these numbers, which is 10485.14 for the given example.
```
$ cat f1.txt 
8
53
3.14
84
73e2
100
2937
```
Read the documentation for glob.glob() and write a program to list all files ending with .txt in the current directory as well as sub-directories, recursively.

Python Comprehensions and Generator expressions

Sundeep — Mon, 01 Feb 2021 15:32:59 +0000

This chapter will show how to use comprehensions and generator expressions for map, filter and reduce operations. You'll also learn about iterators and the yield statement.

Comprehensions

As mentioned earlier, Python provides map() and filter() built-in functions. Comprehensions provide a terser and a faster (usually) way to implement them. However, the syntax can take a while to understand and get comfortable with.

The minimal requirement for a comprehension is a mapping expression (which could include a function call) and a loop. Here's an example:

>>> nums = (321, 1, 1, 0, 5.3, 2)

# manual implementation
>>> sqr_nums = []
>>> for n in nums:
...     sqr_nums.append(n * n)
... 
>>> sqr_nums
[103041, 1, 1, 0, 28.09, 4]

# list comprehension
>>> [n * n for n in nums]
[103041, 1, 1, 0, 28.09, 4]

The general form of the above list comprehension is [expr loop]. Comparing with the manual implementation, the difference is that append() is automatically performed, which is where most of the performance benefit comes from. Note that list comprehension is defined based on the output being a list, input to the for loop can be any iterable (like tuple in the above example).

Here's an example with filtering operation. Instead of the following implementations:

# manual implementation
def remove_dunder(obj):
    names = []
    for n in dir(obj):
        if '__' not in n:
            names.append(n)
    return names

# using 'filter' function
def remove_dunder(obj):
    return list(filter(lambda n: '__' not in n, dir(obj)))

You can use comprehension syntax like this:

>>> def remove_dunder(obj):
...     return [n for n in dir(obj) if '__' not in n]
... 
>>> remove_dunder(dict)
['clear', 'copy', 'fromkeys', 'get', 'items', 'keys', 'pop',
 'popitem', 'setdefault', 'update', 'values']

The general form of the above comprehension is [expr loop condition]. If you can write the manual implementation, it is easy to derive the comprehension version. Put the expression (the argument passed to append() method) first, and then put the loops and conditions in the same order as the manual implementation. With practice, you'll be able to read and write the comprehension versions naturally.

Here's an example with zip() function:

>>> p = [1, 3, 5]
>>> q = [3, 214, 53]
>>> [i + j for i, j in zip(p, q)]
[4, 217, 58]
>>> [i * j for i, j in zip(p, q)]
[3, 642, 265]

And here's a nested loop example:

>>> names = ['Jo', 'Joe', 'Jon']
>>> pairs = []
>>> for i, n1 in enumerate(names):
...     for n2 in names[i+1:]:
...         pairs.append((n1, n2))
... 
>>> pairs
[('Jo', 'Joe'), ('Jo', 'Jon'), ('Joe', 'Jon')]
# note that the loop order is same as the manual implementation
>>> [(n1, n2) for i, n1 in enumerate(names) for n2 in names[i+1:]]
[('Jo', 'Joe'), ('Jo', 'Jon'), ('Joe', 'Jon')]

Similarly, you can build dict and set comprehensions by using {} instead of [] characters. Comprehension syntax inside () characters becomes a generator expression (discussed later in this chapter), so you'll need to use tuple() for tuple comprehension. You can use list(), dict() and set() instead of [] and {} respectively as well.

>>> marks = dict(Rahul=68, Ravi=92, Rohit=75, Rajan=85, Ram=80)
>>> {k: v for k, v in marks.items() if v >= 80}
{'Ravi': 92, 'Rajan': 85, 'Ram': 80}

>>> colors = {'teal', 'blue', 'green', 'yellow', 'red', 'orange'}
>>> {c for c in colors if 'o' in c}
{'yellow', 'orange'}

>>> dishes = ('Poha', 'Aloo tikki', 'Baati', 'Khichdi', 'Makki roti')
>>> tuple(d for d in dishes if len(d) < 6)
('Poha', 'Baati')

If you are still confused with comprehension syntax, see:

Iterator

Partial quote from docs.python glossary: iterator:

An object representing a stream of data. Repeated calls to the iterator’s __next__() method (or passing it to the built-in function next()) return successive items in the stream. When no more data are available a StopIteration exception is raised instead.

The filter() example in the previous section required further processing, such as passing to the list() function to get the output as a list object. This is because the filter() function returns an object that behaves like an iterator. You can pass iterators anywhere iterables are allowed, such as the for loop. Here's an example:

>>> filter_obj = filter(lambda n: '__' not in n, dir(tuple))
>>> filter_obj
<filter object at 0x7fd910e2de80>
>>> for x in filter_obj:
...     print(x)
... 
count
index

One of the differences between an iterable and an iterator is that you can iterate over iterables any number of times (quite the tongue twister, if I may say so myself). Also, the next() function can be used on an iterator, but not iterables. Once you have exhausted an iterator, any attempt to get another item (such as next() or for loop) will result in a StopIteration exception. Iterators are lazy and memory efficient since the results are evaluated only when needed, instead of lying around in a container.

>>> names = filter(lambda n: '__' not in n, dir(tuple))
>>> next(names)
'count'
>>> next(names)
'index'
>>> next(names)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

You can convert an iterable to an iterator using the iter() built-in function.

>>> nums = [321, 1, 1, 0, 5.3, 2]
>>> iter(nums)
<list_iterator object at 0x7fd90e7f8ee0>

Here's a practical example to get a random item from a list without repetition:

>>> import random 
>>> names = ['Jo', 'Ravi', 'Joe', 'Raj', 'Jon']
>>> random.shuffle(names)
>>> random_name = iter(names)
>>> next(random_name)
'Jon'
>>> next(random_name)
'Ravi'

yield

Functions that use yield statement instead of return to create an iterator are known as generators. Quoting from docs.python: Generators:

Each time next() is called on it, the generator resumes where it left off (it remembers all the data values and which statement was last executed).

Here's a fibonacci generator:

>>> def fibonacci(n):
...     a, b = 0, 1
...     for _ in range(n):
...         yield a
...         a, b = b, a + b
... 
>>> fibonacci(5)
<generator object fibonacci at 0x7fd90e7b22e0>
>>> list(fibonacci(10))
[0, 1, 1, 2, 3, 5, 8, 13, 21, 34]

For a more detailed discussion and related features, see:

Generator expressions

Comprehension syntax inside () characters creates an iterator, known as generator expressions. Using a generator expression is memory efficient and faster than comprehensions whenever you need a single use iterable. If you use comprehension, you'll be wasting memory to save the values in a container, only to be discarded once they are processed by a reduce operation such as the sum() function in the below examples.

>>> nums = [100, 53, 32, 0, 11, 5, 2]
>>> (n * n for n in nums)
<generator object <genexpr> at 0x7fd90e7b22e0>
>>> g = (n * n for n in nums)
>>> next(g)
10000

# here's a generator version of the sum_sqr_evens(iterable) function
# note that () is optional here for the generator expression
>>> sum(n * n for n in nums if n % 2 == 0)
11028

# inner product
>>> sum(i * j for i, j in zip((1, 3, 5), (2, 4, 6)))
44

Here's an example with join() method:

>>> items = (1, 'hi', [10, 20], 'bye')
>>> ':'.join(items)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: sequence item 0: expected str instance, int found
>>> ':'.join(str(i) for i in items)
'1:hi:[10, 20]:bye'

Exercises

Write a function that returns a dictionary sorted by values in ascending order.

>>> marks = dict(Rahul=86, Ravi=92, Rohit=75, Rajan=79, Ram=92)
>>> sort_by_value(marks)
{'Rohit': 75, 'Rajan': 79, 'Rahul': 86, 'Ravi': 92, 'Ram': 92}

Write a function that returns a list of string slices as per the following rules:
- return the input string as the only element if its length is less than 3 characters
- otherwise, return all slices that have 2 or more characters
```
>>> word_slices('i')
['i']
>>> word_slices('to')
['to']
>>> word_slices('table')
['ta', 'tab', 'tabl', 'table', 'ab', 'abl', 'able', 'bl', 'ble', 'le']
```
Square even numbers and cube odd numbers. For example, [321, 1, -4, 0, 5, 2] should give you [33076161, 1, 16, 0, 125, 4] as the output.
Calculate sum of squares of the numbers, only if the square value is less than 50. Output for (7.1, 1, -4, 8, 5.1, 12) should be 43.01.

Python Text Processing

Sundeep — Sat, 30 Jan 2021 15:24:12 +0000

This chapter will discuss str methods and introduce a few examples with the string and re modules.

join

The join() method is similar to what the print() function does with the sep option, except that you get a str object as the result. The iterable you pass to join() can only have string elements. On the other hand, print() uses an object's __str__() method to get its string representation (__repr__() method is used as a fallback).

>>> print(1, 2)
1 2
>>> ' '.join((1, 2))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: sequence item 0: expected str instance, int found
>>> ' '.join(('1', '2'))
'1 2'

>>> c = ' :: '
>>> c.join(['This', 'is', 'a', 'sample', 'string'])
'This :: is :: a :: sample :: string'

As an exercise, check what happens if you pass multiple string values separated by comma to join() instead of an iterable.

Transliteration

The translate() method accepts a table of codepoints (numerical value of a character) mapped to another character/codepoint or None (if the character has to be deleted). You can use the ord() built-in function to get the codepoint of characters. Or, you can use the str.maketrans() method to generate the mapping for you.

>>> ord('a')
97
>>> ord('A')
65

>>> str.maketrans('aeiou', 'AEIOU')
{97: 65, 101: 69, 105: 73, 111: 79, 117: 85}

>>> greeting = 'have a nice day'
>>> greeting.translate(str.maketrans('aeiou', 'AEIOU'))
'hAvE A nIcE dAy'

The string module has a collection of constants that are often useful in text processing. Here's an example of deleting punctuation characters.

>>> import string
>>> string.punctuation
'!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~'

>>> para = '"Hi", there! How *are* you? All fine here.'
>>> para.translate(str.maketrans('', '', string.punctuation))
'Hi there How are you All fine here'

>>> chars_to_delete = ''.join(set(string.punctuation) - set('.!?'))
>>> para.translate(str.maketrans('', '', chars_to_delete))
'Hi there! How are you? All fine here.'

As an exercise, read the documentation for features covered in this section. See also stackoverflow: character translation examples.

Removing leading/trailing characters

The strip() method removes consecutive characters from the start/end of the given string. By default it removes whitespace characters, which you can change by passing a str argument. You can use lstrip() and rstrip() methods to work only on the leading and trailing characters respectively.

>>> greeting = '  \t\r\n have    a  nice \t day  \f\v\r\t\n '
>>> greeting.strip()
'have    a  nice \t day'
>>> greeting.lstrip()
'have    a  nice \t day  \x0c\x0b\r\t\n '
>>> greeting.rstrip()
'  \t\r\n have    a  nice \t day'

>>> '"Hi",'.strip(string.punctuation)
'Hi'

The removeprefix() and removesuffix() methods will delete a substring from the start/end of the input string.

>>> 'spare'.removeprefix('sp')
'are'
>>> 'free'.removesuffix('e')
'fre'

# difference between remove and strip
>>> 'cared'.removesuffix('de')
'cared'
>>> 'cared'.rstrip('de')
'car'

Dealing with case

Here's five different methods for changing the case of characters. Word level transformation is determined by consecutive occurrences of alphabets, not limited to separation by whitespace characters.

>>> sentence = 'thIs iS a saMple StrIng'

>>> sentence.capitalize()
'This is a sample string'

>>> sentence.title()
'This Is A Sample String'

>>> sentence.lower()
'this is a sample string'

>>> sentence.upper()
'THIS IS A SAMPLE STRING'

>>> sentence.swapcase()
'THiS Is A SAmPLE sTRiNG'

The string.capwords() method is similar to title() but also allows a specific word separator (whose default is whitespace).

>>> phrase = 'this-IS-a:colon:separated,PHRASE'

>>> phrase.title()
'This-Is-A:Colon:Separated,Phrase'
>>> string.capwords(phrase, ':')
'This-is-a:Colon:Separated,phrase'

is methods

The islower(), isupper() and istitle() methods check if the given string conforms to the specific case pattern. Characters other than alphabets do not influence the result, but at least one alphabet needs to be present for a True output.

>>> 'αλεπού'.islower()
True

>>> '123'.isupper()
False
>>> 'ABC123'.isupper()
True

>>> 'Today is Sunny'.istitle()
False

Here's some examples with isnumeric() and isascii() methods. As an exercise, read the documentation for the rest of the is methods.

# checks if string has numeric characters only, at least one
>>> '153'.isnumeric()
True
>>> ''.isnumeric()
False
>>> '1.2'.isnumeric()
False
>>> '-1'.isnumeric()
False

# False if any character codepoint is outside the range 0x00 to 0x7F
>>> '123—456'.isascii()
False
>>> 'happy learning!'.isascii()
True

Substring and count

The in operator checks if the LHS string is a substring of the RHS string.

>>> sentence = 'This is a sample string'

>>> 'is a' in sentence
True
>>> 'this' in sentence
False
>>> 'this' in sentence.lower()
True
>>> 'test' not in sentence
True

The count() method gives the number of times the given substring is present (non-overlapping).

>>> sentence = 'This is a sample string'
>>> sentence.count('is')
2
>>> sentence.count('w')
0

>>> word = 'phototonic'
>>> word.count('oto')
1

Match start/end

The startswith() and endswith() methods check for the presence of substrings only at the start/end of the input string.

>>> sentence = 'This is a sample string'

>>> sentence.startswith('This')
True
>>> sentence.startswith('is')
False

>>> sentence.endswith('ing')
True
>>> sentence.endswith('ly')
False

If you need to check for multiple substrings, pass a tuple argument.

>>> words = ['refuse', 'impossible', 'present', 'read']
>>> prefix = ('im', 're')
>>> for w in words:
...     if w.startswith(prefix):
...         print(w)
... 
refuse
impossible
read

split

The split() method splits a string based on the given substring and returns a list. By default, whitespace is used for splitting. You can also control the number of splits.

>>> greeting = '  \t\r\n have    a  nice \t day  \f\v\r\t\n '
# note that leading/trailing whitespaces do not create empty elements
>>> greeting.split()
['have', 'a', 'nice', 'day']

# note that the empty elements are preserved here
>>> ':car::jeep::'.split(':')
['', 'car', '', 'jeep', '', '']

>>> 'apple<=>grape<=>mango<=>fig'.split('<=>', maxsplit=1)
['apple', 'grape<=>mango<=>fig']

replace

Use replace() method for substitution operation. Optional third argument allows you to specify number of replacements to be made.

>>> phrase = '2 be or not 2 be'

>>> phrase.replace('2', 'to')
'to be or not to be'

>>> phrase.replace('2', 'to', 1)
'to be or not 2 be'

# recall that string is immutable, you'll need to re-assign if needed
>>> phrase = phrase.replace('2', 'to')
>>> phrase
'to be or not to be'

re module

Regular Expressions is a versatile tool for text processing. You'll find them included as part of standard library of most programming languages that are used for scripting purposes. If not, you can usually find a third-party library. Syntax and features of regular expressions vary from language to language though. re module is the built-in library for Python.

What's so special about regular expressions and why would you need it? It is a mini programming language in itself, specialized for text processing. Parts of a regular expression can be saved for future use, analogous to variables and functions. There are ways to perform AND, OR, NOT conditionals. Operations similar to range() function, string repetition operator and so on. Here's some common use cases:

Sanitizing a string to ensure that it satisfies a known set of rules. For example, to check if a given string matches password rules.
Filtering or extracting portions on an abstract level like alphabets, numbers, punctuation and so on.
Qualified string replacement. For example, at the start or the end of a string, only whole words, based on surrounding text, etc.

>>> import re

# extract non-colon character sequences
>>> ip = ':car::jeep::'
>>> ip.split(':')
['', 'car', '', 'jeep', '', '']
>>> re.findall(r'[^:]+', ip)
['car', 'jeep']

# replace only whole words 'par' and 'hand' with 'X'
# \b is an anchor to restrict the matching to the start/end of words
>>> ip = 'par spare part hand handy unhanded'
>>> re.sub(r'\b(par|hand)\b', 'X', ip)
'X spare part X handy unhanded'

See my book Python re(gex)? for a detailed guide on regular expressions (it is longer than this book!). The book covers the third-party regex module as well.

Exercises

Write a function that checks if two strings are anagrams irrespective of case (assume input is made up of alphabets only).
```
>>> anagram('god', 'Dog')
True
>>> anagram('beat', 'table')
False
>>> anagram('Beat', 'abet')
True
```

Read the documentation and implement these formatting examples with equivalent str methods.

>>> fruit = 'apple'

>>> f'{fruit:=>10}'
'=====apple'
>>> f'{fruit:=<10}'
'apple====='
>>> f'{fruit:=^10}'
'==apple==='

>>> f'{fruit:^10}'
'  apple   '

Write a function that returns a list of words present in the input string.

>>> words('"Hi", there! How *are* you? All fine here.')
['Hi', 'there', 'How', 'are', 'you', 'All', 'fine', 'here']
>>> words('This-Is-A:Colon:Separated,Phrase')
['This', 'Is', 'A', 'Colon', 'Separated', 'Phrase']

Python Sets

Sundeep — Sun, 24 Jan 2021 13:28:02 +0000

set is a mutable, unordered collection of objects. frozenset is similar to set, but immutable. See docs.python: set, frozenset for more details.

Initialization

Sets are declared as a collection of objects separated by a comma within {} curly brace characters. The set() function can be used to initialize an empty set and to convert iterables.

>>> empty_set = set()
>>> empty_set
set()

>>> nums = {-0.1, 3, 2, -5, 7, 1, 6.3, 5}
# note that the order is not the same as declaration
>>> nums
{-0.1, 1, 2, 3, 5, 6.3, 7, -5}

# duplicates are automatically removed
>>> set([3, 2, 11, 3, 5, 13, 2])
{2, 3, 5, 11, 13}
>>> set('initialize')
{'a', 'n', 't', 'l', 'e', 'i', 'z'}

set doesn't allow mutable objects as elements.

>>> {1, 3, [1, 2], 4}
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'list'

>>> {1, 3, (1, 2), 4}
{3, 1, (1, 2), 4}

Set methods and operations

The in operator checks if a value is present in the given set. Since set uses hashtable (similar to dict keys), the lookup time is constant and much faster than ordered collections like list or tuple for large data sets.

>>> colors = {'red', 'blue', 'green'}
>>> 'blue' in colors
True
>>> 'orange' in colors
False

Here's some examples for set operations like union, intersection, etc. You can either use methods or operators, both will give you a new set object instead of in-place modification. The difference is that these methods can accept any iterable, not restricted to set objects.

>>> color_1 = {'teal', 'light blue', 'green', 'yellow'}
>>> color_2 = {'light blue', 'black', 'dark green', 'yellow'}

# union of two sets: color_1 | color_2
>>> color_1.union(color_2)
{'light blue', 'green', 'dark green', 'black', 'teal', 'yellow'}

# common items: color_1 & color_2
>>> color_1.intersection(color_2)
{'light blue', 'yellow'}

# items from color_1 not present in color_2: color_1 - color_2
>>> color_1.difference(color_2)
{'teal', 'green'}
# items from color_2 not present in color_1: color_2 - color_1
>>> color_2.difference(color_1)
{'dark green', 'black'}

# items present in one of the sets, but not both
# i.e. union of above two operations: color_1 ^ color_2
>>> color_1.symmetric_difference(color_2)
{'green', 'dark green', 'black', 'teal'}

As mentioned in Dict chapter, methods like keys(), values() and items() return a set-like object. You can apply set operators on them.

>>> marks_1 = dict(Rahul=86, Ravi=92, Rohit=75)
>>> marks_2 = dict(Jo=89, Rohit=78, Joe=75, Ravi=100)

>>> marks_1.keys() & marks_2.keys()
{'Ravi', 'Rohit'}
>>> marks_1.keys() - marks_2.keys()
{'Rahul'}

Methods like add(), update(), symmetric_difference_update(), intersection_update() and difference_update() will do the modifications in-place.

>>> color_1 = {'teal', 'light blue', 'green', 'yellow'}
>>> color_2 = {'light blue', 'black', 'dark green', 'yellow'}

# union
>>> color_1.update(color_2)
>>> color_1
{'light blue', 'green', 'dark green', 'black', 'teal', 'yellow'}

# adding a single value
>>> color_2.add('orange')
>>> color_2
{'black', 'yellow', 'dark green', 'light blue', 'orange'}

The pop() method will return a random element being removed. Use the remove() method if you want to delete an element based on its value. The discard() method is similar to remove(), but it will not generate an error if the element doesn't exist. The clear() method will delete all the elements.

>>> colors = {'red', 'blue', 'green'}

>>> colors.pop()
'blue'
>>> colors
{'green', 'red'}

>>> colors.clear()
>>> colors
set()

Here's some examples for comparison operations.

>>> names_1 = {'Ravi', 'Rohit'}
>>> names_2 = {'Ravi', 'Ram', 'Rohit', 'Raj'}

>>> names_1 == names_2
False

# same as: names_1 <= names_2
>>> names_1.issubset(names_2)
True

# same as: names_2 >= names_1
>>> names_2.issuperset(names_1)
True

# disjoint means there's no common elements: not names_1 & names_2
>>> names_1.isdisjoint(names_2)
False
>>> names_1.isdisjoint({'Jo', 'Joe'})
True

Exercises

Write a function that checks whether an iterable has duplicate values or not.
```
>>> has_duplicates('pip')
True
>>> has_duplicates((3, 2))
False
```
What does the above function return for has_duplicates([3, 2, 3.0])?

Python Dictionaries

Sundeep — Sat, 23 Jan 2021 13:35:25 +0000

Dictionaries can be thought of as a collection of key-value pairs or a named list of items. It used to be unordered, but recent Python versions ensure that the insertion order is maintained.

Initialization and accessing elements

A dict data type is declared within {} characters and each item requires two values — an immutable data type for key, followed by : character and finally a value of any data type. The elements are separated by a comma character, just like the other container types.

To access an element, the syntax is dict_variable[key]. Retrieving an item takes a constant amount of time, irrespective of the size of the dict (see Hashtables for details). Dictionaries are mutable, so you can change an item's value, add items, remove items, etc.

>>> marks = {'Rahul': 86, 'Ravi': 92, 'Rohit': 75, 'Rajan': 79}

>>> marks['Rohit']
75
>>> marks['Rahul'] += 5
>>> marks['Ram'] = 67 
>>> del marks['Rohit']

# note that the insertion order is maintained
>>> marks
{'Rahul': 91, 'Ravi': 92, 'Rajan': 79, 'Ram': 67}

Here's an example with list and tuple keys.

>>> list_key = {[1, 2]: 42}
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'list'

>>> items = {('car', 2): 'honda', ('car', 5): 'tesla', ('bike', 10): 'hero'}
>>> items[('bike', 10)]
'hero'

You can also use the dict() function for initialization in various ways. If all the keys are of str data type, you can use the same syntax as keyword arguments seen earlier with function definitions. You can also pass a container type having two values per element, such as a list of tuples as shown below.

>>> marks = dict(Rahul=86, Ravi=92, Rohit=75, Rajan=79)
>>> marks
{'Rahul': 86, 'Ravi': 92, 'Rohit': 75, 'Rajan': 79}

>>> items = [('jeep', 20), ('car', 3), ('cycle', 5)]
>>> dict(items)
{'jeep': 20, 'car': 3, 'cycle': 5}

Another way to initialize is to use the fromkeys() method that accepts an iterable and an optional value (default is None). The same value will be assigned to all the keys, so be careful if you want to use a mutable object, since the same reference will be used as well.

>>> colors = ('red', 'blue', 'green')

>>> dict.fromkeys(colors)
{'red': None, 'blue': None, 'green': None}
>>> dict.fromkeys(colors, 255)
{'red': 255, 'blue': 255, 'green': 255}

get and setdefault

If you try to access a dict key that doesn't exist, you'll get a KeyError exception. If you do not want an exception to occur, you can use the get() method. By default it'll return a None value for keys that do not exist, which you can change by providing a default value as the second argument.

>>> marks = dict(Rahul=86, Ravi=92, Rohit=75, Rajan=79)

>>> marks['Ron']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'Ron'

>>> marks.get('Ravi')
92
>>> value = marks.get('Ron')
>>> print(value)
None
>>> marks.get('Ron', 0)
0

Here's a more practical example:

>>> vehicles = ['car', 'jeep', 'car', 'bike', 'bus', 'car', 'bike']
>>> hist = {}
>>> for v in vehicles:
...     hist[v] = hist.get(v, 0) + 1
... 
>>> hist
{'car': 3, 'jeep': 1, 'bike': 2, 'bus': 1}

Using get() method will not automatically add keys that do not exist yet to the dict object. You can use setdefault() method, which behaves similarly to get() except that keys will get created if not found. See also docs.python: collections.defaultdict.

>>> marks = dict(Rahul=86, Ravi=92, Rohit=75, Rajan=79)

>>> marks.get('Ram', 40)
40
>>> marks
{'Rahul': 86, 'Ravi': 92, 'Rohit': 75, 'Rajan': 79}

>>> marks.setdefault('Ram', 40)
40
>>> marks
{'Rahul': 86, 'Ravi': 92, 'Rohit': 75, 'Rajan': 79, 'Ram': 40}

Iteration

The default for loop over a dict object will give you a key for each iteration.

>>> fruits = dict(banana=12, papaya=5, mango=10, fig=100)

>>> for k in fruits:
...     print(f'{k}:{fruits[k]}')
... 
banana:12
papaya:5
mango:10
fig:100

# you'll also get only the keys if you apply list(), tuple() or set()
>>> list(fruits)
['banana', 'papaya', 'mango', 'fig']

As an exercise,

given fruits dictionary as defined in the above code snippet, what do you think will happen when you use a, *b, c = fruits?
given nums = [1, 4, 6, 22, 3, 5, 4, 3, 6, 2, 1, 51, 3, 1], keep only first occurrences of a value from this list without changing the order of elements. You can do it with dict features presented so far. [1, 4, 6, 22, 3, 5, 2, 51] should be the output. See Using dict to eliminate duplicates while retaining order if you are not able to solve it.

Dict methods and operations

The in operator checks if a key is present in the given dictionary. The keys() method returns all the keys and values() method returns all the values. These methods return a custom set-like object, but with insertion order maintained.

>>> marks = dict(Rahul=86, Ravi=92, Rohit=75, Rajan=79)

>>> 'Ravi' in marks
True
>>> 'Ram' in marks
False

>>> marks.keys()
dict_keys(['Rahul', 'Ravi', 'Rohit', 'Rajan'])

>>> marks.values()
dict_values([86, 92, 75, 79])

The items() method can be used to get a key-value tuple for each iteration.

>>> fruits = dict(banana=12, papaya=5, mango=10, fig=100)

# set-like object
>>> fruits.items()
dict_items([('banana', 12), ('papaya', 5), ('mango', 10), ('fig', 100)])

>>> for fruit, qty in fruits.items():
...     print(f'{fruit}\t: {qty}')
... 
banana  : 12
papaya  : 5
mango   : 10
fig     : 100

The del statement example seen earlier removes the given key without returning the value associated with it. You can use the pop() method to get the value as well. The popitem() method removes the last added item and returns the key-value pair as a tuple.

>>> marks = dict(Rahul=86, Ravi=92, Rohit=75, Rajan=79)

>>> marks.pop('Ravi')
92
>>> marks
{'Rahul': 86, 'Rohit': 75, 'Rajan': 79}

>>> marks.popitem()
('Rajan', 79)
>>> marks
{'Rahul': 86, 'Rohit': 75}

The update() method allows you to add/update items from another dictionary or a container with key-value pair elements.

>>> marks = dict(Rahul=86, Ravi=92, Rohit=75, Rajan=79)
>>> marks.update(dict(Jo=89, Joe=75, Ravi=100))
# note that 'Ravi' has '100' as the updated value
>>> marks
{'Rahul': 86, 'Ravi': 100, 'Rohit': 75, 'Rajan': 79, 'Jo': 89, 'Joe': 75}

>>> fruits = dict(banana=12, papaya=5, mango=10, fig=100)
>>> fruits.update([('onion', 3), ('pea', 10)])
>>> fruits
{'banana': 12, 'papaya': 5, 'mango': 10, 'fig': 100, 'onion': 3, 'pea': 10}

The | operator is similar to the update() method, except that you get a new dict object instead of in-place modification.

>>> d1 = {'banana': 12, 'papaya': 5, 'mango': 20}
>>> d2 = {'mango': 10, 'fig': 100}

# this used to be done using unpacking, i.e. {**d1, **d2}
>>> d1 | d2
{'banana': 12, 'papaya': 5, 'mango': 10, 'fig': 100}

Arbitrary keyword arguments

To accept arbitrary number of keyword arguments, use **var_name in the function definition. This has to be declared the last, after all the other types of arguments. Idiomatically, **kwargs is used as the variable name. See stackoverflow: Decorators demystified for a practical use case of arbitrary keyword arguments.

>>> def many(**kwargs):
...     print(f'{kwargs = }')
... 
>>> many()
kwargs = {}
>>> many(num=5)
kwargs = {'num': 5}
>>> many(car=5, jeep=25)
kwargs = {'car': 5, 'jeep': 25}

Turning it around, when you have a function defined with keyword arguments, you can unpack a dictionary while calling the function.

>>> def greeting(phrase='hello', style='='):
...     print(f'{phrase:{style}^{len(phrase)+6}}')
... 
>>> greeting()
===hello===
>>> d = {'style': '-', 'phrase': 'have a nice day'}
>>> greeting(**d)
--------have a nice day---

Mutable effect in Python

Sundeep — Fri, 22 Jan 2021 13:18:43 +0000

int, float, str and tuple are immutable data types. On the other hand, types like list and dict are mutable. This chapter will discuss what happens when you pass a variable to a function or when you assign them to another value/variable.

id

The id() built-in function returns the identity (reference) of an object. Here's some examples to show what happens when you assign a variable to another value/variable.

>>> num1 = 5
>>> id(num1)
140204812958128
# here, num1 gets a new identity
>>> num1 = 10
>>> id(num1)
140204812958288

# num2 will have the same reference as num1
>>> num2 = num1
>>> id(num2)
140204812958288

# num2 gets a new reference, num1 won't be affected
>>> num2 = 4
>>> id(num2)
140204812958096
>>> num1
10

Pass by reference

Variables in Python store references to an object, not their values. When you pass a list object to a function, you are passing the reference to this object. Since list is mutable, any in-place changes made to this object within the function will also be reflected in the original variable that was passed to the function. Here's an example:

>>> def rotate(ip):
...     ip.insert(0, ip.pop())
... 
>>> nums = [321, 1, 1, 0, 5.3, 2]
>>> rotate(nums)
>>> nums
[2, 321, 1, 1, 0, 5.3]

This is true even for slices of a sequence containing mutable objects. Also, as shown in the example below, tuple doesn't prevent mutable elements from being changed.

>>> nums_2d = ([1, 3, 2, 10], [1.2, -0.2, 0, 2], [100, 200])
>>> last_two = nums_2d[-2:]

>>> last_two[0][-1] = 'apple'
>>> last_two[1][-1] = 'ball'

>>> last_two
([1.2, -0.2, 0, 'apple'], [100, 'ball'])
>>> nums_2d
([1, 3, 2, 10], [1.2, -0.2, 0, 'apple'], [100, 'ball'])

As an exercise, use id() function to verify that the identity of last two elements of nums_2d variable in the above example is the same as the identity of both the elements of last_two variable.

Slicing notation copy

If you wish to copy whole/part of a list object such that changing the copy version doesn't affect the original list, the solution will depend on the presence of mutable elements.

Here's an example where all the elements are immutable. In this case, using slice notation is safe for copying.

>>> items = [3, 'apple', 100.23, 'fig']
>>> items_copy = items[:]

>>> id(items)
140204765864256
>>> id(items_copy)
140204765771968

# the individual elements will still have the same reference
>>> id(items[0]) == id(items_copy[0])
True

>>> items_copy[0] += 1000
>>> items_copy
[1003, 'apple', 100.23, 'fig']
>>> items
[3, 'apple', 100.23, 'fig']

On the other hand, if the list has mutable objects, using slice notation won't stop the copy from modifying the original.

>>> nums_2d = [[1, 3, 2, 10], [1.2, -0.2, 0, 2], [100, 200]]
>>> nums_2d_copy = nums_2d[:]

>>> nums_2d_copy[0][0] = 'oops'

>>> nums_2d_copy
[['oops', 3, 2, 10], [1.2, -0.2, 0, 2], [100, 200]]
>>> nums_2d
[['oops', 3, 2, 10], [1.2, -0.2, 0, 2], [100, 200]]

copy.deepcopy

The copy built-in module has a deepcopy() method if you wish to recursively create new copy of all the elements of a mutable object.

>>> import copy

>>> nums_2d = [[1, 3, 2, 10], [1.2, -0.2, 0, 2], [100, 200]]
>>> nums_2d_deepcopy = copy.deepcopy(nums_2d)

>>> nums_2d_deepcopy[0][0] = 'yay'

>>> nums_2d_deepcopy
[['yay', 3, 2, 10], [1.2, -0.2, 0, 2], [100, 200]]
>>> nums_2d
[[1, 3, 2, 10], [1.2, -0.2, 0, 2], [100, 200]]

As an exercise, create a deepcopy of only the first two elements of nums_2d object from the above example.

Python Lists

Sundeep — Thu, 21 Jan 2021 14:12:53 +0000

List is a container data type, similar to tuple but mutable. Lists are typically used to store and manipulate ordered collection of values.

Tuple and Sequence operations chapter is a prerequisite for this one.

Initialization and Slicing

Lists are declared as a comma separated values within [] square brackets. Unlike tuple there's no ambiguity in using [] characters, so there's no special requirement of trailing comma for a single element list object. You can use a trailing comma if you wish, which is helpful to easily change a list declared across multiple lines.

# 1D example
>>> vowels = ['a', 'e', 'i', 'o', 'u']
>>> vowels[0]
'a'
# same as vowels[4] since len(vowels) - 1 = 4
>>> vowels[-1]
'u'

# 2D example
>>> student = ['learnbyexample', 2021, ['Linux', 'Vim', 'Python']]
>>> student[1]
2021
>>> student[2]
['Linux', 'Vim', 'Python']
>>> student[2][-1]
'Python'

Since list is a mutable data type, you can modify the object after initialization. You can either change a single element or multiple elements using slicing notation.

>>> nums = [1, 4, 6, 22, 3, 5]

>>> nums[0] = 100
>>> nums
[100, 4, 6, 22, 3, 5]

>>> nums[-3:] = [-1, -2, -3]
>>> nums
[100, 4, 6, -1, -2, -3]

# list will automatically shrink/expand as needed
>>> nums[1:4] = [2000]
>>> nums
[100, 2000, -2, -3]
>>> nums[1:2] = [3.14, 4.13, 6.78]
>>> nums
[100, 3.14, 4.13, 6.78, -2, -3]

List methods and operations

This section will discuss some of the list methods and operations. See docs.python: list methods for documentation. As mentioned earlier, you can use dir(list) to view the available methods of an object.

Use the append() method to add a single element to the end of a list object. If you need to append multiple items, you can pass an iterable to the extend() method. As an exercise, check what happens if you pass an iterable to the append() method and a non-iterable value to the extend() method. What happens if you pass multiple values to both these methods?

>>> books = []
>>> books.append('Cradle')
>>> books.append('Mistborn')
>>> books
['Cradle', 'Mistborn']

>>> items = [3, 'apple', 100.23]
>>> items.extend([4, 'mango'])
>>> items
[3, 'apple', 100.23, 4, 'mango']
>>> items.extend((-1, -2))
>>> items.extend(range(3))
>>> items.extend('hi')
>>> items
[3, 'apple', 100.23, 4, 'mango', -1, -2, 0, 1, 2, 'h', 'i']

The count() method will give the number of times a value is present.

>>> nums = [1, 4, 6, 22, 3, 5, 2, 1, 51, 3, 1]
>>> nums.count(3)
2
>>> nums.count(31)
0

The index() method will give the index of the first occurrence of a value. As seen with tuple, this method will raise ValueError if the value isn't present.

>>> nums = [1, 4, 6, 22, 3, 5, 2, 1, 51, 3, 1]

>>> nums.index(3)
4

The pop() method removes the last element of a list by default. You can pass an index to delete that specific item and the list will be automatically re-arranged. Return value is the element being deleted.

>>> primes = [2, 3, 5, 7, 11]
>>> last = primes.pop()
>>> last
11
>>> primes
[2, 3, 5, 7]
>>> primes.pop(2)
5
>>> primes
[2, 3, 7]

>>> student = ['learnbyexample', 2021, ['Linux', 'Vim', 'Python']]
>>> student.pop(1)
2021
>>> student[-1].pop(1)
'Vim'
>>> student
['learnbyexample', ['Linux', 'Python']]
>>> student.pop()
['Linux', 'Python']
>>> student
['learnbyexample']

To remove multiple elements using slicing notation, use the del statement. Unlike the pop() method, there is no return value.

>>> nums = [1.2, -0.2, 0, 2, 4, 23]
>>> del nums[0]
>>> nums
[-0.2, 0, 2, 4, 23]
>>> del nums[2:4]
>>> nums
[-0.2, 0, 23]

>>> nums_2d = [[1, 3, 2, 10], [1.2, -0.2, 0, 2], [100, 200]]
>>> del nums_2d[0][1:3]
>>> del nums_2d[1]
>>> nums_2d
[[1, 10], [100, 200]]

The pop() method deletes an element based on its index. Use the remove() method to delete an element based on its value. You'll get ValueError if the value isn't found.

>>> even_numbers = [2, 4, 6, 8, 10]
>>> even_numbers.remove(8)
>>> even_numbers
[2, 4, 6, 10]

The clear() method removes all the elements. You might wonder why not just assign an empty list? If you have observed closely, all of the methods seen so far modified the list object in-place. This is useful if you are passing a list object to a function and expect the function to modify the object itself instead of returning a new object. More on that later.

>>> nums = [1.2, -0.2, 0, 2, 4, 23]
>>> nums.clear()
>>> nums
[]

You've already seen how to add element(s) at the end of a list using append() and extend() methods. The insert() method is the opposite of pop() method. You can provide a value to be inserted at the given index. As an exercise, check what happens if you pass a list value. Also, what happens if you pass more than one value?

>>> books = ['Sourdough', 'Sherlock Holmes', 'To Kill a Mocking Bird']
>>> books.insert(2, 'The Martian')
>>> books
['Sourdough', 'Sherlock Holmes', 'The Martian', 'To Kill a Mocking Bird']

The reverse() method reverses a list in-place. If you use slicing notation, you'll get a new object.

>>> primes = [2, 3, 5, 7, 11]
>>> primes.reverse()
>>> primes
[11, 7, 5, 3, 2]

>>> primes[::-1]
[2, 3, 5, 7, 11]
>>> primes
[11, 7, 5, 3, 2]

Here's some examples with comparison operators. Quoting from documentation:

For two collections to compare equal, they must be of the same type, have the same length, and each pair of corresponding elements must compare equal (for example, [1,2] == (1,2) is false because the type is not the same).

Collections that support order comparison are ordered the same as their first unequal elements (for example, [1,2,x] <= [1,2,y] has the same value as x <= y). If a corresponding element does not exist, the shorter collection is ordered first (for example, [1,2] < [1,2,3] is true).

>>> primes = [2, 3, 5, 7, 11]
>>> nums = [2, 3, 5, 11, 7]
>>> primes == nums
False
>>> primes == [2, 3, 5, 7, 11]
True

>>> [1, 1000] < [2, 3]
True
>>> [1000, 2] < [1, 2, 3]
False

>>> ['a', 'z'] > ['a', 'x']
True
>>> [1, 2, 3] > [10, 2]
False
>>> [1, 2, 3] > [1, 2]
True

Sorting and company

The sort() method will order the list object in-place. The sorted() built-in function provides the same functionality for iterable types and returns an ordered list.

>>> nums = [1, 5.3, 321, 0, 1, 2]

# ascending order
>>> nums.sort()
>>> nums
[0, 1, 1, 2, 5.3, 321]

# descending order
>>> nums.sort(reverse=True)
>>> nums
[321, 5.3, 2, 1, 1, 0]

>>> sorted('fuliginous')
['f', 'g', 'i', 'i', 'l', 'n', 'o', 's', 'u', 'u']

The key argument accepts the name of a built-in/user-defined function (i.e. function object) for custom sorting. If two elements are deemed equal based on the result of the function, the original order will be maintained (known as stable sorting). Here's some examples:

# based on the absolute value of an element
# note that the input order is maintained for all three values of "4"
>>> sorted([-1, -4, 309, 4.0, 34, 0.2, 4], key=abs)
[0.2, -1, -4, 4.0, 4, 34, 309]

# based on the length of an element
>>> words = ('morello', 'irk', 'fuliginous', 'crusado', 'seam')
>>> sorted(words, key=len, reverse=True)
['fuliginous', 'morello', 'crusado', 'seam', 'irk']

If the custom user-defined function required is just a single expression, you can create anonymous functions with lambda expressions instead of a full-fledged function. As an exercise, read docs.python: Sorting and implement the below examples using operator module instead of lambda expressions.

# based on second element of each item
>>> items = [('car', 20), ('jeep', 3), ('cycle', 5)]
>>> sorted(items, key=lambda e: e[1], reverse=True)
[('car', 20), ('cycle', 5), ('jeep', 3)]

# based on number of words, assuming space as the word separator
>>> dishes = ('Poha', 'Aloo tikki', 'Baati', 'Khichdi', 'Makki roti')
>>> sorted(dishes, key=lambda s: s.count(' '), reverse=True)
['Aloo tikki', 'Makki roti', 'Poha', 'Baati', 'Khichdi']

You can use sequence types like list or tuple to specify multiple sorting conditions. Make sure to read the sequence comparison examples from previous section before trying to understand the following examples.

>>> dishes = ('Poha', 'Aloo tikki', 'Baati', 'Khichdi', 'Makki roti')

# word-count and dish-names, both descending order
>>> sorted(dishes, key=lambda s: (s.count(' '), s), reverse=True)
['Makki roti', 'Aloo tikki', 'Poha', 'Khichdi', 'Baati']

# word-count descending order, dish-names ascending order
# the main trick is to negate the numerical value
>>> sorted(dishes, key=lambda s: (-s.count(' '), s))
['Aloo tikki', 'Makki roti', 'Baati', 'Khichdi', 'Poha']

As an exercise, given nums = [1, 4, 5, 2, 51, 3, 6, 22], determine and implement the sorting condition based on the required output shown below:

[4, 2, 6, 22, 1, 5, 51, 3]
[2, 4, 6, 22, 1, 3, 5, 51]
[22, 6, 4, 2, 51, 5, 3, 1]

Here's some examples with min() and max() functions.

>>> nums = [321, 0.5, 899.232, 5.3, 2, 1, -1]
>>> min(nums)
-1
>>> max(nums)
899.232
>>> min(nums, key=abs)
0.5

Random items

You have already seen a few examples with random module in earlier chapters. This section will show a few examples with methods that act on sequence data types.

First up, getting a random element from a non-empty sequence using the choice() method.

>>> import random

>>> random.choice([4, 5, 2, 76])
76
>>> random.choice('hello')
'e'

The shuffle() method randomizes the elements of a list in-place.

>>> items = ['car', 20, 3, 'jeep', -3.14, 'hi']

>>> random.shuffle(items)
>>> items
['car', 3, -3.14, 'jeep', 'hi', 20]

Use the sample() method to get a list of specified number of random elements. As an exercise, see what happens if you pass a slice size greater than the number of elements present in the input sequence.

>>> random.sample((4, 5, 2, 76), k=3)
[4, 76, 2]

>>> random.sample(range(1000), k=5)
[490, 26, 9, 745, 919]

Map, Filter and Reduce

Many operations on container objects can be defined in terms of these three concepts. For example, if you want to sum the square of all even numbers:

separating out even numbers is Filter (i.e. only elements that satisfy a condition are retained)
square of such numbers is Map (i.e. each element is transformed by a mapping function)
final sum is Reduce (i.e. you get one value out of multiple values)

One or more of these operations may be absent depending on the problem statement. A function for the first of these steps could look like:

>>> def get_evens(iter):
...     op = []
...     for n in iter:
...         if n % 2 == 0:
...             op.append(n)
...     return op
... 
>>> get_evens([100, 53, 32, 0, 11, 5, 2])
[100, 32, 0, 2]

Function after the second step could be:

>>> def sqr_evens(iter):
...     op = []
...     for n in iter:
...         if n % 2 == 0:
...             op.append(n * n)
...     return op
... 
>>> sqr_evens([100, 53, 32, 0, 11, 5, 2])
[10000, 1024, 0, 4]

And finally, the function after the third step could be:

>>> def sum_sqr_evens(iter):
...     total = 0
...     for n in iter:
...         if n % 2 == 0:
...             total += n * n
...     return total
... 
>>> sum_sqr_evens([100, 53, 32, 0, 11, 5, 2])
11028

Here's some examples with sum(), all() and any() built-in reduce functions.

>>> sum([321, 0.5, 899.232, 5.3, 2, 1, -1])
1228.032

>>> conditions = [True, False, True]
>>> all(conditions)
False
>>> any(conditions)
True
>>> conditions[1] = True
>>> all(conditions)
True

>>> nums = [321, 1, 1, 0, 5.3, 2]
>>> all(nums)
False
>>> any(nums)
True

Python also provides map(), filter() and functools.reduce() for such problems. But, see Comprehensions and Generator expressions chapter before deciding to use them.

Exercises

Write a function that returns the product of a sequence of numbers. Empty sequence or sequence containing non-numerical values should raise TypeError.
- product([-4, 2.3e12, 77.23, 982, 0b101]) should give -3.48863356e+18
- product(range(2, 6)) should give 120
- product(()) and product(['a', 'b']) should raise TypeError

Write a function that removes dunder names from dir() output.

>>> remove_dunder(list)
['append', 'clear', 'copy', 'count', 'extend', 'index',
 'insert', 'pop', 'remove', 'reverse', 'sort']
>>> remove_dunder(tuple)
['count', 'index']

Tuple and Sequence operations

Sundeep — Wed, 20 Jan 2021 13:17:37 +0000

This chapter will discuss the tuple data type and some of the common sequence operations. Data types like str, range, list and tuple fall under Sequence types. Binary Sequence Types aren't discussed in this book. Some of the operations behave differently or do not apply for certain types, see docs.python: Common Sequence Operations for details.

See also docs.python: collections for specialized container data types.

Sequences and iterables

Quoting from docs.python glossary: sequence:

An iterable which supports efficient element access using integer indices via the __getitem__() special method and defines a __len__() method that returns the length of the sequence. Some built-in sequence types are list, str, tuple, and bytes. Note that dict also supports __getitem__() and __len__(), but is considered a mapping rather than a sequence because the lookups use arbitrary immutable keys rather than integers.

Quoting from docs.python glossary: iterable:

An object capable of returning its members one at a time. Examples of iterables include all sequence types (such as list, str, and tuple) and some non-sequence types like dict, file objects...

Initialization

Tuples are declared as a collection of zero or more objects, separated by a comma within () parentheses characters. Each element can be specified as a value by itself or as an expression. The outer parentheses are optional if comma separation is present. Here's some examples:

# can also use: empty_tuple = tuple()
>>> empty_tuple = ()
>>> type(empty_tuple)
<class 'tuple'>

# note the trailing comma, otherwise it will result in a 'str' data type
# same as 'apple', since parentheses are optional here
>>> one_element = ('apple',)

# multiple elements example
>>> dishes = ('Aloo tikki', 'Baati', 'Khichdi', 'Makki roti', 'Poha')

# mixed data type example, uses expressions as well
>>> mixed = (1+2, 'two', (-3, -4), empty_tuple)
>>> mixed
(3, 'two', (-3, -4), ())

You can use the tuple() built-in function to convert from an iterable data type to tuple. Here's some examples:

>>> chars = tuple('hello')
>>> chars
('h', 'e', 'l', 'l', 'o')

>>> tuple(range(3, 10, 3))
(3, 6, 9)

Tuples are immutable, but individual elements can be either mutable or immutable. As an exercise, given chars = tuple('hello'), see what's the output of the expression chars[0] and the statement chars[0] = 'H'.

Slicing

One or more elements can be retrieved from a sequence using the slicing notation (this wouldn't work for an iterable like dict or set). It works similarly to the start/stop/step logic seen with the range() function. The default step is 1. Default value for start and stop depends on whether the step is positive or negative.

>>> primes = (2, 3, 5, 7, 11)

# index starts with 0
>>> primes[0]
2

# start=2 and stop=4, default step=1
# note that the element at index 4 (stop value) isn't part of the output
>>> primes[2:4]
(5, 7)
# default start=0
>>> primes[:3]
(2, 3, 5)
# default stop=len(seq)
>>> primes[3:]
(7, 11)

# copy of the sequence, same as primes[::1]
>>> primes[:]
(2, 3, 5, 7, 11)

You can use negative index to get elements from the end of the sequence. seq[-n] is equivalent to seq[len(seq) - n].

>>> primes = (2, 3, 5, 7, 11)

# len(primes) - 1 = 4, so this is same as primes[4]
>>> primes[-1]
11

# seq[-n:] will give the last n elements
>>> primes[-1:]
(11,)
>>> primes[-2:]
(7, 11)

Here's some examples with different step values.

>>> primes = (2, 3, 5, 7, 11)

# same as primes[0:5:2]
>>> primes[::2]
(2, 5, 11)

# retrieve elements in reverse direction
# note that the element at index 1 (stop value) isn't part of the output
>>> primes[3:1:-1]
(7, 5)
# reversed sequence
# would help you with the palindrome exercise from Control structures chapter
>>> primes[::-1]
(11, 7, 5, 3, 2)

As an exercise, given primes = (2, 3, 5, 7, 11),

what happens if you use primes[5] or primes[-6]?
what happens if you use primes[:5] or primes[-6:]?
is it possible to get the same output as primes[::-1] by using an explicit number for stop value? If not, why not?

Sequence unpacking

You can map the individual elements of an iterable to multiple variables. This is known as sequence unpacking and it is handy in many situations.

>>> details = ('2018-10-25', 'car', 2346)
>>> purchase_date, vehicle, qty = details
>>> purchase_date
'2018-10-25'
>>> vehicle
'car'
>>> qty
2346

Here's how you can easily swap variable values.

>>> num1 = 3.14
>>> num2 = 42
>>> num3 = -100

# RHS is a single tuple data type (recall that parentheses are optional)
>>> num1, num2, num3 = num3, num1, num2
>>> print(f'{num1 = }; {num2 = }; {num3 = }')
num1 = -100; num2 = 3.14; num3 = 42

Unpacking isn't limited to single value assignments. You can use a * prefix to assign all the remaining values, if any is left, to a list variable.

>>> values = ('first', 6.2, -3, 500, 'last')

>>> x, *y = values
>>> x
'first'
>>> y
[6.2, -3, 500, 'last']

>>> a, *b, c = values
>>> a
'first'
>>> b
[6.2, -3, 500]
>>> c
'last'

As an exercise, what do you think will happen for these cases, given nums = (1, 2):

a, b, c = nums
a, *b, c = nums
*a, *b = nums

Returning multiple values

Tuples are also the preferred way to return multiple values from a function. Here's some examples:

>>> def min_max(iter):
...     return min(iter), max(iter)
... 
>>> min_max('visualization')
('a', 'z')
>>> small, big = min_max((-3, 10, -42, 53.2))
>>> small
-42
>>> big
53.2

The min_max(iter) user-defined function above returns both the minimum and maximum values of a given iterable input. min() and max() are built-in functions. You can either save the output as a tuple or unpack into multiple variables. You'll see built-in functions that return tuple as output later in this chapter.

The use of both min() and max() in the above example is for illustration purpose only. As an exercise, write a custom logic that iterates only once over the input sequence and calculates both minimum/maximum simultaneously.

Iteration

You have already seen examples with for loop that iterates over a sequence data type. Here's a refresher:

>>> nums = (3, 6, 9)
>>> for n in nums:
...     print(f'square of {n} is {n ** 2}')
... 
square of 3 is 9
square of 6 is 36
square of 9 is 81

In the above example, you get one element per each iteration. If you need the index of the elements as well, you can use the enumerate() built-in function. You'll get a tuple value per each iteration, containing index (starting with 0 by default) and the value at that index. Here's some examples:

>>> nums = (42, 3.14, -2)

>>> for t in enumerate(nums):
...     print(t)
... 
(0, 42)
(1, 3.14)
(2, -2)

>>> for idx, val in enumerate(nums):
...     print(f'{idx}: {val:>5}')
... 
0:    42
1:  3.14
2:    -2

The enumerate() built-in function has a start=0 default valued argument. As an exercise, change the above snippet to start the index from 1 instead of 0.

Arbitrary number of arguments

As seen before, the print() function can accept zero or more values separated by a comma. Here's a portion of the documentation as a refresher:

print(value, ..., sep=' ', end='\n', file=sys.stdout, flush=False)

You can write your own functions to accept arbitrary number of arguments as well. The syntax is similar to the sequence unpacking examples seen earlier in the chapter. A * prefix to an argument name will allow it to accept zero or more values. Such an argument will be packed as a tuple data type and it should always be specified after positional arguments (if any). Idiomatically, args is used as the tuple variable name. Here's an example:

>>> def many(a, *args):
...     print(f'{a = }; {args = }')
... 
>>> many()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: many() missing 1 required positional argument: 'a'
>>> many(1)
a = 1; args = ()
>>> many(1, 'two', 3)
a = 1; args = ('two', 3)

Here's a more practical example:

>>> def sum_nums(*args):
...     total = 0
...     for n in args:
...         total += n
...     return total
... 
>>> sum_nums()
0
>>> sum_nums(3, -8)
-5
>>> sum_nums(1, 2, 3, 4, 5)
15

As an exercise,

add a default valued argument initial which should be used to initialize total instead of 0 in the above sum_nums() function. For example, sum_nums(3, -8) should give -5 and sum_nums(1, 2, 3, 4, 5, initial=5) should give 20.
what would happen if you call the above function like sum_nums(initial=5, 2)?
what would happen if you have nums = (1, 2) and call the above function like sum_nums(*nums, total=3)?
in what ways does this function differ from the sum() built-in function?

See also docs.python: Arbitrary Argument Lists.

Section Arbitrary keyword arguments will discuss how to define functions that accept arbitrary number of keyword arguments.

zip

Use zip() to iterate over two or more iterables simultaneously. Every iteration, you'll get a tuple with an item from each of the iterables. Iteration will stop when any of the input iterables is exhausted, use itertools.zip_longest() if you want to go on until the longest iterable is exhausted.

Here's an example:

>>> odd = (1, 3, 5)
>>> even = (2, 4, 6)
>>> for i, j in zip(odd, even):
...     print(i + j)
... 
3
7
11

As an exercise, write a function that returns the sum of product of corresponding elements of two sequences (for example, the result should be 44 for (1, 3, 5) and (2, 4, 6)).

Tuple methods

While this book won't discuss Object-Oriented Programming (OOP) in any detail, you'll still see plenty examples for using them. You've already seen a few examples with modules. See Practical Python Programming and Fluent Python if you want to learn about Python OOP in depth. See also docs.python: Data model.

Data types in Python are all internally implemented as classes. You can use the dir() built-in function to get a list of valid attributes for an object.

# you can also use tuple objects such as 'odd' and 'even' declared earlier
>>> dir(tuple)
['__add__', '__class__', '__class_getitem__', '__contains__', '__delattr__',
 '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__',
 '__getitem__', '__getnewargs__', '__gt__', '__hash__', '__init__',
 '__init_subclass__', '__iter__', '__le__', '__len__', '__lt__', '__mul__',
 '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__rmul__',
 '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'count', 'index']

>>> even = (2, 4, 6)
# same as: len(even)
>>> even.__len__()
3

The non-dunder names (last two items) in the above listing will be discussed in this section. But first, a refresher on the in membership operator is shown below.

>>> num = 5
>>> num in (10, 21, 33)
False

>>> num = 21
>>> num in (10, 21, 33)
True

The count() method returns the number of times a value is present in the tuple object.

>>> nums = (1, 4, 6, 22, 3, 5, 2, 1, 51, 3, 1)
>>> nums.count(3)
2
>>> nums.count(31)
0

The index() method will give the index of the first occurrence of a value. It will raise ValueError if the value isn't present, which you can avoid by using the in operator first. Or, you can use the try-except statement to handle the exception as needed.

>>> nums = (1, 4, 6, 22, 3, 5, 2, 1, 51, 3, 1)

>>> nums.index(3)
4

>>> n = 31
>>> nums.index(n)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: tuple.index(x): x not in tuple
>>> if n in nums:
...     print(nums.index(n))
... else:
...     print(f'{n} not present in "nums" tuple')
... 
31 not present in "nums" tuple

The list and str sequence types have many more methods and they will be discussed separately in later chapters.

Testing

Sundeep — Tue, 19 Jan 2021 13:34:33 +0000

Testing can only prove the presence of bugs, not their absence. — Edsger W. Dijkstra

There, it should work now. — All programmers

Above quotes chosen from this collection at softwareengineering.stackexchange.

General tips

Another crucial aspect in the programming journey is knowing how to write tests. In bigger projects, usually there are separate engineers (often in much larger number than developers) to test the code. Even in those cases, writing a few sanity test cases yourself can help you develop faster knowing that the changes aren't breaking basic functionality.

There's no single consensus on test methodologies. There is Unit testing, Integration testing, Test-driven development (TDD) and so on. Often, a combination of these is used. These days, machine learning is also being considered to reduce the testing time, see Testing Firefox more efficiently with machine learning for an example.

When I start a project, I usually try to write the programs incrementally. Say I need to iterate over files from a directory. I will make sure that portion is working (usually with print() statements), then add another feature — say file reading and test that and so on. This reduces the burden of testing a large program at once at the end. And depending upon the nature of the program, I'll add a few sanity tests at the end. For example, for my command_help project, I copy pasted a few test runs of the program with different options and arguments into a separate file and wrote a program to perform these tests programmatically whenever the source code is modified.

assert

For simple cases, the assert statement is good enough. If the expression passed to assert evaluates to False, the AssertionError exception will be raised. You can optionally pass a message, separated by a comma after the expression to be tested. See docs.python: assert for documentation.

# passing case
>>> assert 2 < 3

# failing case
>>> num = -2
>>> assert num >= 0, 'only positive integer allowed'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AssertionError: only positive integer allowed

Here's a sample program (solution for one of the exercises from Control structures chapter).

# nested_braces.py
def max_nested_braces(expr):
    max_count = count = 0
    for char in expr:
        if char == '{':
            count += 1
            if count > max_count:
                max_count = count
        elif char == '}':
            if count == 0:
                return -1
            count -= 1

    if count != 0:
        return -1
    return max_count

def test_cases():
    assert max_nested_braces('a*b') == 0
    assert max_nested_braces('a*b+{}') == 1
    assert max_nested_braces('a*{b+c}') == 1
    assert max_nested_braces('{a+2}*{b+c}') == 1
    assert max_nested_braces('a*{b+c*{e*3.14}}') == 2
    assert max_nested_braces('{{a+2}*{b+c}+e}') == 2
    assert max_nested_braces('{{a+2}*{b+{c*d}}+e}') == 3
    assert max_nested_braces('{{a+2}*{{b+{c*d}}+e*d}}') == 4
    assert max_nested_braces('a*b{') == -1
    assert max_nested_braces('a*{b+c}}') == -1
    assert max_nested_braces('}a+b{') == -1
    assert max_nested_braces('a*{b+c*{e*3.14}}}') == -1
    assert max_nested_braces('{{a+2}*{{b}+{c*d}}+e*d}}') == -1

if __name__ == '__main__':
    test_cases()
    print('all tests passed')

max_count = count = 0 is a terse way to initialize multiple variables to the same value. Okay to use for immutable types (will be discussed later) like int, float and str.

If everything goes right, you should see the following output.

$ python3.9 nested_braces.py
all tests passed

As an exercise, randomly change the logic of max_nested_braces function and see if any of the tests fail.

assert statements can be skipped by using python3.9 -O

Writing tests helps you in many ways. It could help you guard against typos and accidental editing. Often, you'll need to tweak a program in future to correct some bugs or add a feature — tests would again help to give you confidence that you haven't messed up already working cases. Another use case is refactoring, where you rewrite a portion of the program (sometimes entire) without changing its functionality.

Here's an alternate implementation of max_nested_braces(expr) function from the above program using regular expressions.

# nested_braces_re.py
# only the function is shown below
import re

def max_nested_braces(expr):
    count = 0
    while True:
        expr, no_of_subs = re.subn(r'\{[^{}]*\}', '', expr)
        if no_of_subs == 0:
            break
        count += 1

    if re.search(r'[{}]', expr):
        return -1
    return count

pytest

For large projects, simple assert statements aren't enough to adequately write and manage tests. You'll require built-in module unittest or popular third-party modules like pytest. See python test automation frameworks for more resources.

This section will show a few introductory examples with pytest. If you visit a project on PyPI, the pytest page for example, you can copy the installation command as shown in the image below. You can also check out the statistics link (https://libraries.io/pypi/pytest for example) as a minimal sanity check that you are installing the correct module.

# virtual environment
$ pip install pytest

# normal environment
$ python3.9 -m pip install --user pytest

After installation, you'll have pytest usable as a command line application by itself. The two programs discussed in the previous section can be run without any modification as shown below. This is because pytest will automatically use function names starting with test for its purpose. See doc.pytest: Conventions for Python test discovery for full details.

# -v is verbose option, use -q for quiet version
$ pytest -v nested_braces.py
=================== test session starts ====================
platform linux -- Python 3.9.0, pytest-6.2.1, py-1.10.0,
    pluggy-0.13.1 -- /usr/local/bin/python3.9
cachedir: .pytest_cache
rootdir: /home/learnbyexample/Python/programs
collected 1 item                                           

nested_braces.py::test_cases PASSED                  [100%]

==================== 1 passed in 0.03s =====================

Here's an example where pytest is imported as well.

# exception_testing.py
import pytest

def sum2nums(n1, n2):
    types_allowed = (int, float)
    assert type(n1) in types_allowed, 'only int/float allowed'
    assert type(n2) in types_allowed, 'only int/float allowed'
    return n1 + n2

def test_valid_values():
    assert sum2nums(3, -2) == 1
    # see https://stackoverflow.com/q/5595425
    from math import isclose
    assert isclose(sum2nums(-3.14, 2), -1.14)

def test_exception():
    with pytest.raises(AssertionError) as e:
        sum2nums('hi', 3)
    assert 'only int/float allowed' in str(e.value)

    with pytest.raises(AssertionError) as e:
        sum2nums(3.14, 'a')
    assert 'only int/float allowed' in str(e.value)

pytest.raises() allows you to check if exceptions are raised for the given test cases. You can optionally check the error message as well. The with context manager will be discussed in a later chapter. Note that the above program doesn't actually call any executable code, since pytest will automatically run the test functions.

$ pytest -v exception_testing.py
=================== test session starts ====================
platform linux -- Python 3.9.0, pytest-6.2.1, py-1.10.0,
    pluggy-0.13.1 -- /usr/local/bin/python3.9
cachedir: .pytest_cache
rootdir: /home/learnbyexample/Python/programs
collected 2 items                                          

exception_testing.py::test_valid_values PASSED       [ 50%]
exception_testing.py::test_exception PASSED          [100%]

==================== 2 passed in 0.02s =====================

The above illustrations are trivial examples. Here's some advanced learning resources:

realpython: Getting started with testing in Python
pytest — calmcode video series and Testing Python Applications
obeythetestinggoat — TDD for the Web, with Python, Selenium, Django, JavaScript and pals
testdriven: Modern Test-Driven Development in Python — TDD guide and has a real world application example
Serious Python — deployment, scalability, testing, and more

Forem: Sundeep

I wrote a short, introductory book for Python

Ebook links

Web version

Python Introduction

Sundeep ・ Jan 8 ・ 7 min read

GitHub repo

Table of Contents

Motivation and FAQ

Feedback

CLI apps with Python

sys.argv

In-place editing with fileinput

argparse

Accepting stdin

Executing external commands from Python

Using os module

subprocess.run

shell=True

Changing shell

Capture output

Dealing with files in Python

open and close

Context manager

read, readline and readlines

write

File processing modules

Exercises

Python Comprehensions and Generator expressions

Comprehensions

Iterator

yield

Generator expressions

Exercises

Python Text Processing

join

Transliteration

Removing leading/trailing characters

Dealing with case

is methods

Substring and count

Match start/end

split

replace

re module

Exercises

Python Sets

Initialization

Set methods and operations

Exercises

Python Dictionaries

Initialization and accessing elements

get and setdefault

Iteration

Dict methods and operations

Arbitrary keyword arguments

Mutable effect in Python

id

Pass by reference

Slicing notation copy

copy.deepcopy

Python Lists

Initialization and Slicing

List methods and operations

Sorting and company

Random items

Map, Filter and Reduce

Exercises

Tuple and Sequence operations

Sequences and iterables

Initialization

Slicing

Sequence unpacking

Returning multiple values

Iteration

Arbitrary number of arguments

zip

Tuple methods

Testing

General tips