Forem: Shiva Gaire

A Yogi's Guide to Debug Python Programs

Shiva Gaire — Thu, 14 Sep 2023 18:28:39 +0000

There are many resources on how to write code but not many on how to debug. In this article, I am highlighting my approach to debug both synchronous and asynchronous Python programs.

Approach to debugging

Using IDE:

Running a program in debug mode in an IDE like PyCharm, or Vscode is the easiest way to debug for someone who loves debugging using IDE-assisted features. It provides information regarding the state of objects and available attributes for objects which makes debugging easier. You can set a breakpoint at any given line in your IDE and start debugging and jumping steps repeatedly until the bug is found and fixed.

The print statements or `logging` module

When I started programming, I mostly used good old print statements and/or the logging module to know the state of the program in each line. I used to insert random print statements to debug the problems, with funny print or logging statements.

The best debugging tool is still careful thought, coupled with judiciously placed print statements.

- Brian W. Kernighan

Rubber Duck

The gist of this debugging approach is to verbally explain the problem that you are facing to someone else, it may be a rubber duck or a colleague(if you are lucky). While explaining the problems to others, our understanding also gets better which will help in connect the dots required for solving problems.

REPL

REPL(Read, Evaluate, Print, and Loop), or the way of providing Python statements directly to the interpreter console to evaluate the result. This approach saves time as you can just evaluate the Python statements rather than executing the whole Python file. REPL is mostly useful when dealing with standard modules or just trying to find the results of common data types related functions and the results. REPL to evaluate the results of standard modules/datatypes is a convenient approach to understanding the behavior of underlying operations.

Using dir to look up all attributes available for modules, objects, and data types is still my favorite thing. The REPL below shows the use of dir to string module and set data type.

>>> import string
>>> dir(string)
['Formatter', 'Template', '_ChainMap', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '_re', '_sentinel_dict', '_string', 'ascii_letters', 'ascii_lowercase', 'ascii_uppercase', 'capwords', 'digits', 'hexdigits', 'octdigits', 'printable', 'punctuation', 'whitespace']
>>> a={1,2,3}
>>> dir(a)
['__and__', '__class__', '__class_getitem__', '__contains__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__iand__', '__init__', '__init_subclass__', '__ior__', '__isub__', '__iter__', '__ixor__', '__le__', '__len__', '__lt__', '__ne__', '__new__', '__or__', '__rand__', '__reduce__', '__reduce_ex__', '__repr__', '__ror__', '__rsub__', '__rxor__', '__setattr__', '__sizeof__', '__str__', '__sub__', '__subclasshook__', '__xor__', 'add', 'clear', 'copy', 'difference', 'difference_update', 'discard', 'intersection', 'intersection_update', 'isdisjoint', 'issubset', 'issuperset', 'pop', 'remove', 'symmetric_difference', 'symmetric_difference_update', 'union', 'update']
>>> b={3,4,5}
>>> a.intersection(b)
{3}

The Python debugger or `pdb`

This is something that I have used primarily in my software engineering career to debug Python programs. The module pdb ( breakpoint in recent Python versions) temporarily stops the execution of a program and lets you interact with the states of a program. You can insert a breakpoint on any line and move over to the next statement to find and fix problems. Combining pdb prompt with dir is a match made in heaven when it comes to debugging. The Python debugger(pdb) has a set of commands like n,c, l that you can refer here to use within the debugging prompt.

❯ python test_requests.py
> /tmp/test_requests.py(6)<module>()
-> print(response.text)
(Pdb) l
  1     import requests
  2
  3     response = requests.get('https://example.org/')
  4
  5     import pdb; pdb.set_trace()
  6  -> print(response.text)
[EOF]
(Pdb) dir(response)
['__attrs__', '__bool__', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__enter__', '__eq__', '__exit__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__iter__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__nonzero__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_content', '_content_consumed', '_next', 'apparent_encoding', 'close', 'connection', 'content', 'cookies', 'elapsed', 'encoding', 'headers', 'history', 'is_permanent_redirect', 'is_redirect', 'iter_content', 'iter_lines', 'json', 'links', 'next', 'ok', 'raise_for_status', 'raw', 'reason', 'request', 'status_code', 'text', 'url']
(Pdb) response.headers
{'Content-Encoding': 'gzip', 'Accept-Ranges': 'bytes', 'Age': '411952', 'Cache-Control': 'max-age=604800', 'Content-Type': 'text/html; charset=UTF-8', 'Date': 'Sun, 10 Sep 2023 19:57:02 GMT', 'Etag': '"3147526947+gzip"', 'Expires': 'Sun, 17 Sep 2023 19:57:02 GMT', 'Last-Modified': 'Thu, 17 Oct 2019 07:18:26 GMT', 'Server': 'ECS (nyb/1D07)', 'Vary': 'Accept-Encoding', 'X-Cache': 'HIT', 'Content-Length': '648'}
(Pdb)

The pdbpp or ipdb Python packages available in PyPI to enhances the debugging experience further.

Traceback module

I have used the standard traceback module to figure out the sequence of functions and their order of execution leading to the exception. It aids in debugging by displaying detailed information on the call stack, line numbers, and source of error. It is useful when the exception is handled without exposing many details of the error.

import requests
import traceback

def print_response():
    try:
        response = requests.get('https://thisdoesnot.exist/')
    except Exception as e:
       traceback.print_exc()
       print("Request failed")
       return
    print(response.text)

def main():
    print("inside main")
    print_response()

main()

AI-Assisted debugging

LLM tools like Chat GPT, GitHub copilot, Codium AI, etc. are quite good at explaining and even generating code. It can be leveraged while debugging as it can sometimes provide valuable insights into the bug or issue faced.

Debugging Asynchronous Python Programs

Debugging synchronous programs is hard, but debugging asynchronous programs is harder.

As mentioned in python documentation, we can enable asyncio debug mode for easier debugging of asynchronous programs.

Ways to enable asyncio debug mode:

Setting the PYTHONASYNCIODEBUG environment variable to 1.
Using the Python development mode with python -x dev or by setting the environment variable PYTHONDEVMODE to 1
Passing debug=True to asyncio.run().
Calling loop.set_debug() when the instance of loop is available.

These are the benefits of using asyncio debug mode:

Finds not awaited co-routines and logs them
Shows execution time of coroutine or I/O selector
Shows callbacks that take more than 100ms by default. We can also change this time by using loop.slow_callback_duration to define the minimum seconds to consider slow callbacks

In addition to the above debug mode, there are tools like aiomonitor , which inspects the asyncio loop and provides debugging capabilities. This exposes the telnet server to provide REPL capabilities, to inspect the state of asyncio application. This can also be used while debugging async programs within a docker container or in a remote server.

Whatever the bug, debugging always requires mental clarity just like a yogi. Don't stress and happy debugging!! 🐍

References:

Feel free to comment and If you learned something from this article, please share it with your friends.

You may also follow me on LinkedIn and Twitter .

Beginners guide to Iterator, Iteration and Iterable in Python

Shiva Gaire — Sat, 29 May 2021 12:25:22 +0000

Introduction

In programming iterable, iterator and iteration are pretty familiar words and one can have much confusion explaining and understanding these pretty common topics. Let's demystify them one by one.

Iterable

The natural language definition of Iterable is capable of being repeated or iterated.
However, the pythonic definition of iterator is an object that implements these two magic methods: __iter__ and __getitem__.

The __iter__ method of iterable returns an iterator, which we will see next and similarly the __getitem__ method takes an argument or key item from its sequences starting index zero and until the index is no longer valid and python raises an exception called IndexError. The __getitem__ method is used mostly for index-based lookup over the collections.

In python default types: list & strings have__iter__ & __getitem__ implemented and hence we can refer them as an iterable.

For a more concise visual representation let's assume a single row bookshelf where there are 7 different books.

On this bookshelf, we can go through each book one by one, also we can change order of the books, can swap the places, hence we can consider this bookshelf as an object. But as a reader, I can read only one book at a time.
So, it can be clearly seen this bookshelf can be considered iterable from what we have discussed to this point.

Iterator

In programming, an iterator is an object that let us travel through the Iterable objects. This iterator object remembers where it is during the process of travelling.

In python, an iterator is an object with a __next__ method implemented in it. The
__next__ method returns the next value while traversing(travelling through) the iterable object sequence starting from index zero of the iterable. After each call, the __next__ method updates its state to point to the next object in the iterable sequence. And when the travelling process is completed, it raises the StopError exception.

For a visual representation, let's assume you have a toy laser light that you can use to point to the book sequentially to read through them one by one, during this process to take out the first book you first point your toy laser light to the first book in the bookshelf, you take out that book and while doing so the laser focus points to next book in the sequence and so on until going through them one by one completes.

>>> bookshelf = ['The Diary of a young girl', 'Crime and punishment', 'Romeo and Juliet', ...] #7 books
>>> flashlight  = iter(bookshelf)
>>> next(flashlight)
The Diary of a young girl
>>> next(flashlight)
Crime and punishment

In python, the for loop, map and list comprehension automatically call this next method.

Iteration

The Merriam Webster dictionary defines iteration as:

the repetition of a sequence of computer instructions a specified number of times or until a condition is met— compare RECURSION
one execution of a sequence of operations or instructions in an iteration.

From the above bookshelf analogy, we can say the process of repeating through book one by one can be referred to as an iteration.

In programming, while doing iteration the same block of code is repeated every time. We use loops to achieve iteration that basically executes instruction or sequence of instruction repeatedly.

Pythonic Conclusion

Iterable: list, strings, and any objects that have implemented __getitem__ and __iter__ methods.

Iterator: Use of iter() function on an iterable gives iterator object, Iterator has next method in it. This iterator object remember where it is during the iteration

Iteration: The use of for loop while loop or even list comprehension.
Please leave an example in the comment box :D

Reference

Finally, If you learned something from this article, please share it with your friends.

You may also follow me on LinkedIn and Twitter .

Multiple inheritance and MRO in python

Shiva Gaire — Mon, 24 May 2021 06:13:29 +0000

Introduction

Let's first talk about the word inheritance, oxford dictionary defines it as something from the past or from your family that affects the way you behave, look, etc.

In object-oriented programming, inheritance facilitates code reuse with the primary motivation of reusing or extending the parent or base class in children or subclass, hence the subclass or children class is affected by the parent class. It however provides freedom to children class to modify the parent's attributes and methods.

Inheritance

Let's see an example which is most common in all paradigm, here Children class inherits from Parent class and inherits some methods of Parent class and also overrides the Parent class implementation:

class Parent():
    def parent_function(self):
        print("parent_function")

    def common_function(self):
        print("Parent's common_function")


class Children(Parent):
    def child_function(self):
        print("Child's child_function")

    def common_function(self):
        print("Child's common_function")

if __name__ == "__main__":
    child = Children()
    parent = Parent()

    parent.parent_function() # #prints 'parent_function'

    child.child_function()  #prints 'child's function'
    child.parent_function() #prints 'parent_function'

    # common of both
    parent.common_function() #prints "Parent's common_function"
    child.common_function() #prints "Child's common_function"

This type of single inheritance is common across all Object-Oriented Paradigm, However, Python has something more to offer with multiple inheritance, which means you can inherit from multiple base classes.

Multiple Inheritance

Multiple Inheritance refers to the inheritance that uses two or more base class. This is how you define multiple inheritance using class keyword for multiple baseClass:

class SubClass(baseClass, baseClass2, baseClass3, ....)

Python has its renowned multiple inheritance feature which comes in handy when working with a class that needs implementation from two or more class.

When working with multiple base class there arises confusion if one or more base class implements the same method. Let's see how this is handled in python in next step:

Method Resolution Order

Multiple inheritance can create confusion when there are one or more base classes that implement the same method. Python(2.3 and newer) version introduced the renowned method resolution order or mro to deal with the method resolution while using multiple inheritance.

Method resolution order(MRO) in short has its root in C3 linearization algorithm which basically outlines the basis for MRO list calculation in python multilevel inheritance. Method Resolution Order is the python provided calculation of inheritance graph.

The MRO calculation has 3 major points to consider for the method resolution list from a subclass.

Subclasses comes first rather than the base classes
Base class definition order is preserved
Fits the above two criteria for all MRO calculation in a program

If this is violated then C3 prohibits the python from inheritance declaration.

Let's see an example to see the mro list.

class A():
    def __init__(self):
        print("Hi from A")

class B(A):
    def __init__(self):
        print("Hi From B")

class C(A):
    def __init__(self):
        print("Hi from C")

class D(B, C):
    pass
class E(C, B):
    pass


print(D.__mro__) 
# (<class '__main__.D'>, <class '__main__.B'>, <class '__main__.C'>, <class '__main__.A'>, <class 'object'>)
print(E.__mro__) 
# ((<class '__main__.E'>, <class '__main__.C'>, <class '__main__.B'>, <class '__main__.A'>, <class 'object'>))

We use the __mro__ attribute often pronounced as dun~der mro (double underscore mro) with class name, It gives the tuple of classes defining the method's resolution order for the subclass.

In the above example, we can see the diamond inheritance formation. So, on printing D.__mro__ attribute, it gives:

(<class '__main__.D'>, <class '__main__.B'>, <class '__main__.C'>, <class '__main__.A'>, <class 'object'>)

and on printing E.__mro__ , it gives:

(<class '__main__.E'>, <class '__main__.C'>, <class '__main__.B'>, <class '__main__.A'>, <class 'object'>)

Notice the difference in the second element of the above tuple, its because the base class order is preserved for the class definition of class D and E: D(B, C) and E(C, B).

So, <class '__main__.C'> in the second index for the E subclass and <class '__main__.B'> in the second index for B subclass.

The method resolution order will be entirely based on the above tuple order,

which means when any method is invoked for the instance of class D, the method of class D will be resolved first and if not found it will try to resolve from the second element from the D.__mro__ tuple , which is class B and so on to the next element, if no such method is implemented it will result in AttributeError.

If the implementation is found for the current lookup the search stops right there and the method will be resolved from there.

If you made it upto here, Please come up in discussion to check the __mro__ for the below snippet.

class A():
    def __init__(self):
        print("Hi from A")

class B(A):
    def __init__(self):
        print("Hi From B")

class C(A):
    def __init__(self):
        print("Hi from C")

class D(B, A, C):
    pass

Why?

Finally, If you learned something from this article, please share it with your friends.

You may also follow me on LinkedIn and Twitter

Big O notation and Algorithmic Complexity analysis

Shiva Gaire — Tue, 20 Nov 2018 04:21:52 +0000

Introduction

When executing a particular task using algorithms, one common thing that hits every problem solver brain is the efficient and fast algorithm to solve that problem. But, what do exactly fast and efficient means?

Is it the measure of real processing time?

No, it's not the measure of real-time like second and minutes because the old computer with Pentium processor may take a long time to process the same algorithm than the new Intel i3 processor, or, A bad algorithm written in Assembly may execute faster than a good algorithm written in python.

So, run time of program can't be considered to measure algorithmic efficiency.

Hence, to measure the algorithmic efficiency, the concept of Asymptotic Complexity of a program and a notation for describing this called Big O was introduced. Big O is the worst case, Big Omega and Big Theta are best and average case notations. Worst case means we are mostly unlucky for that problem domain , i.e precondition of task do not favour us. Complexity analysis is a tool that allows us to explain how an algorithm behaves as the input grows larger.

Scenarios

Sorting an array of size 10000 compared to array of size 100 and analyzing how run time of the program grows.

Now, Lets dive deeper,

Counting the number of character in a string:

One simplest approach is traversing through whole string letter by letter and incrementing a counter variable by 1. This algorithmic approach run in linear time with respect to number of character 'n' in the string, i.e, it runs in O(n).

Why ??

Using this approach the time required to traverse the entire string is proportional to number of character in string,
i.e time required to traverse string with 40 character is twice the time required to trverse string with 20 character as the amount of time to look individual character is same.

Another approach is, declaring a variable and storing the number of character in a variable say "length" early in the program, i.e, before storing the very first character. Now, there is no need to look at string, instead one have to check the value of that variable. The accessing of such variable is generally asymptotically constant time operation, or O(1). This is because Asymptotic means "How the run time change as input grows". In this approach the length of string whether it has one character or thousandsof character, the only thing we need to do is to find string length and which can be done by reading the "length" variable and the reading time for this variable is constant regardless of string size. Hence this approach can be referred as running in constant time.

The O(1) does not change with the size of inputs.

Variations

There are many different Big O runtimes like O(n), O(n²) etc

O(n²) are asymptotically slower than O(n) i.e if n grows O(n²) will take more time than O(n).

This dosen't means O(n) always run faster, maybe if input is smaller then O(n²) may work faster yet unnoticed

Similarly, we may have logarithmic O(log(n)) for some cases like in Binary search. Binary search cut the array size in half with each operation.

Simpler Program can be analysed by counting number of nested loop.

A single loop over n items has O(n) complexity.
A loop within a loop has O(n²) complexity

| |
| Big O Variant |

The Original Blog post about this is here.

Forem: Shiva Gaire

A Yogi's Guide to Debug Python Programs

Approach to debugging

Using IDE:

The print statements or logging module

Rubber Duck

REPL

The Python debugger or pdb

Traceback module

AI-Assisted debugging

Debugging Asynchronous Python Programs

Beginners guide to Iterator, Iteration and Iterable in Python

Introduction

Iterable

Iterator

Iteration

Pythonic Conclusion

Reference

Multiple inheritance and MRO in python

Introduction

Inheritance

Multiple Inheritance

Method Resolution Order

Big O notation and Algorithmic Complexity analysis

Introduction

Scenarios

Variations

The print statements or `logging` module

The Python debugger or `pdb`