Forem: Ashutosh Kumar

10 Things You Didn’t Know You Could Do in Python3

Ashutosh Kumar — Fri, 11 Aug 2023 08:18:53 +0000

Photo by Alex Chumak on Unsplash

Python is the third most commonly-used programming language. It has introduced many new features, simplifying and enhancing the development process. Yet, many developers remain unaware of the full potential that Python 3 has to offer.

This article will uncover ten lesser-known and underutilized capabilities in Python (v3.5 to v3.11). These hidden gems can improve your coding efficiency and productivity. So, without further ado, let’s dive into these fascinating Python 3 features.

1. Walrus Operator

It’s a new syntax introduced in Python 3.11 that assigns values to variables as part of a larger expression.

Old way

a = "This is a sample string text"

if len(a) > 10:
    print("Length of string a = ", len(a))

New way

a = "This is a sample string text"

if (len_a := len(a)) > 10: # Note the walrus operator will compute value of len(a) and assign it to variable len_a
    print("Length of string a = ", len_a)

2. Structural Pattern Matching

You might remember reading that Python doesn’t support switch statements because if-else can achieve the same result. You are in for a treat. Python3.10 introduced something similar.

Old way

def http_error(status):
    if status == 400:
        return "Bad request"
    elif status == 404:
        return "Not found"
    elif status == 418:
        return "I'm a teapot"
    else:
        return "Something's wrong with the internet"

New way

def http_error(status):
    match status:
        case 400:
            return "Bad request"
        case 404:
            return "Not found"
        case 418:
            return "I'm a teapot"
        case _: # This is a wildcard operator
            return "Something's wrong with the internet"

3. Merging Dictionaries

Python3.9 introduced a new way of merging or updating dictionaries. It complements the existing dict.update and {**d1, **d2} methods of merging dictionaries.

Old way

x = {"key1": "value1 from x", "key2": "value2 from x"}
y = {"key2": "value2 from y", "key3": "value3 from y"}

print({ **x,** y})
print({ **y,** x}) # Note here keys of y would be given preference over keys of x

New way

x = {"key1": "value1 from x", "key2": "value2 from x"}
y = {"key2": "value2 from y", "key3": "value3 from y"}

# Note the usage of OR (|) operator
print(x | y)
prin(y | x) # Note here keys of y would be given preference over keys of x

4. Removing Prefix or Suffix

Python3.9 introduced dedicated methods to get rid of prefixes or suffixes in strings.

Old way

x = "prefixstring"
y = "stringsuffix"

print(x.split("prefix")[-1])
print(y.split("suffix")[0])

New way

x = "prefixstring"
y = "stringsuffix"

print(x.removeprefix("prefix"))
print(y.removesuffix("suffix"))

5. Positional Only Parameters

Python provides to define positional and keyword arguments for a function. But you can pass them interchangeably. Python3.8 introduced a way to enforce that you cannot pass positional arguments as keyword arguments.

Old way

def fun(pos_arg1, pos_arg2, key_arg_1 = None):
    print("positional arguments: ", pos_arg1, pos_arg2)
    print("Keyword arguments: ", key_arg_1)

# It will work
fun(1, 2, 3) # Passing keyword argument as positional argument

# It will also work
fun(pos_arg1=1, pos_arg2=2, key_arg_1=3) # Passing positional argument as keyword argument

New way

def fun(pos_arg1, pos_arg2, /, key_arg_1 = None):
    print("positional arguments: ", pos_arg1, pos_arg2)
    print("Keyword arguments: ", key_arg_1)

# It will work
fun(1, 2, 3) # Passing keyword argument as positional argument

# It won't work
fun(1, pos_arg2=2, key_arg_1=3) # Passing one positional argument as keyword argument
fun(pos_arg1=1, pos_arg2=2, key_arg_1=3) # Passing both positional arguments as keyword argument

6. Time with nanoseconds Precision

Did you ever need nanosecond precision in getting time to compare code performances? Python3.7 introduced new methods in the time library.

Old way

import time

start = time.time()
end = time.time()

print(end - start)
# Provides sub-second precision, though that precision varies by platform
# >> 2.7179718017578125e-05 Notice the precision is in e-05

New way

import time

start = time.time_ns()
end = time.time_ns()

print(end - start)
# Provides nanosecond precision
# >> 47000 Notice the figure is in nanoseconds

7. f-strings

Python3.6 introduced f-strings to provide an easy way of formatting strings.

Old way

a = 1

print("Value of a = {}".format(a))

New way

a = 1

print(f"Value of a = {a}")

8. Underscores in Numeric Literals

Python 3.6 introduced allowing adding underscores in numeric literals for better readability.

Old way

a = 1000000000000000 # try counting the number of 0's

print(type(a))
# >> <class 'int'>

New way

a = 1_000_000_000_000_000

print(type(a))
# >> <class 'int'>

9. Matrix Multiplication Operator

Python3.5 introduced a dedicated @ infix operator for matrix multiplication.

Old way

import numpy

x = numpy.ones(3)
m = numpy.eye(3)

print(x.dot(m))

New way

import numpy # NumPy 1.10 has support for the new operator:

x = numpy.ones(3)
m = numpy.eye(3)

print(x @ m) # @ is the new matrix matrix-multiplication operator

10. Approximate Equality

Python3.5 introduced dedicated methods to tell whether two values are approximately equal or “close” to each other.

Old way

a = 5.0
b = 4.99998

print(abs(a - b) <= 1e-5)

New way

import math

a = 5.0
b = 4.99998

print(math.isclose(a, b, rel_tol=1e-5))

Resources

You Could Be Wrong About Probability

Ashutosh Kumar — Tue, 08 Aug 2023 13:58:00 +0000

Probability has always fascinated me. It makes the hidden backbone of Machine Learning and Artificial Intelligence. I had the opportunity to study it in school and college. But it wasn't until I took up courses on Bayesian Statistics that I realized how wrong my understanding was about it.

You might have studied probability in school or college. You might have come across the question, "What's the probability of getting heads on tossing a coin"? If your answer is 1/2, think again. It's where it gets interesting.

Mathematics is generally viewed in light of being "consistent". We assume that a problem would always have the same solution no matter how we solve it. It's true except for when it comes to probability. Probability stands like an exception in that it has three different definitions or frameworks. Approaching the same problem with these definitions could yield different (and valid) answers.

To showcase the same, let's consider the following problem. We will solve it using all three frameworks of Probability. One thing common across all the frameworks is that the total probability of all the outcomes of an experiment is always 1.

"My friend Sovit gave me a coin. He didn't tell me if the coin is fair or not. What's the probability of getting heads on this coin?"

Classical Framework

It's the simplest framework in probability. It's also the easiest to understand.

The classical framework says that "Equally likely outcomes have equal probability".

In the above problem, we don't know if the coin is fair. We cannot say if getting heads is equally likely as getting tails. So, we cannot solve this problem using the classical framework.

But to showcase the usage of this framework, let's assume that the coin is fair. It means that getting heads is equally likely as getting tails. Since these are the only two possible outcomes and the total probability is 1, the probability of getting heads is 1/2.

The classical framework might look rudimentary but it's also the most abused framework. Arguments like "Either there's life on Mars or there isn't and so the probability of existence of life on Mars in 1/2" are wrong. Because the classical framework only works when outcomes are equally likely. In this case, the existence and non-existence of life on Mars are not equally likely.

Frequentist Framework

It's one of the most used frameworks in probability. If you have solved any problem in probability, you might have likely used the frequentist framework to do so.

The frequentist framework says that to compute the probability of an event, we need to conduct an experiment and observe the outcome. Repeat the experiment an infinite number of times. And, the probability of the event is P(E) = Count(favorable outcomes) / Count(total outcomes).

In practice, we cannot conduct an experiment an infinite number of times. So, we do it a finitely large number of times. For our problem, let's conduct the experiment 10 times. Let's assume that we got 6 heads and 4 tails. So, the probability of getting heads is 0.6.

The frequentist framework also has limitations. Consider the problem to find the probability of rain tomorrow. By definition, we need to have an infinite number of parallel universes. Then we would need to observe the tomorrow in each of these universes and count the ones where it's raining. But, it's not possible. Besides, why would we compute the probability of rain tomorrow if we can observe tomorrow?

Bayesian Framework

It's one of the most used frameworks in probability. It's also the easiest to understand but difficult to work with.

Bayesian framework says that the probability of an event is what you think it is. It's more about your personal perspective. You are watching cricket and Sachin Tendulkar is at 94. You exclaim there's a 90% chance that he would hit a century. That's your Bayesian probability of the event.

So far in the above two frameworks, we have missed focusing on other key information in the problem: "My friend Sovit gave me the coin". Sovit is my friend and I know him. He has given me other coins in the past. Let's say that those coins had a probability of 0.4 of turning heads. It's called "prior" information. The above two frameworks don't have any way of using it. It's where the Bayesian framework shines. It allows us to use both the prior information and the data, unlike the frequentist framework that relies only on data. We will have to assume how much we trust our prior and how much we trust our data. Let's say we trust both 50% (called weights). The probability of heads then would be a weighted average of prior and data: 0.5 * 0.4 + 0.5 * 0.6 = 0.5.

The Bayesian framework can provide more realistic answers by utilizing prior information. But, we have to make assumptions on weights. This is the critical point of criticism. Since we make assumptions, it is possible to skew the results based on our biases.

So, that's all about probability and it's different frameworks. Let me know in the comments if this blew your mind as it did mine. Give me some claps if you liked the article.

Resources

Courses that I did on Bayesian Statistics: Bayesian Statistics: From Concept to Data Analysis and Bayesian Statistics: Techniques and Models.

How to Write Impeccably Clean Code That Will Save Your Sanity

Ashutosh Kumar — Thu, 27 Jul 2023 19:53:03 +0000

Article published on GitConnected.

Writing code is a skill that anyone can acquire, but attaining the highest standards of clean code requires discipline and dedication. In this article, we will explore practical strategies and tools to help you elevate your Python code to impeccable levels. By gradually incorporating these best practices into your development lifecycle, you can ensure clean, bug-free code that stands the test of time.

Adhering to best practices not only maintains code cleanliness but also minimizes the risk of introducing bugs. While it may require an initial investment of time, following these practices ultimately reduces development effort in the long run. As a seasoned full-stack software engineer, I have consistently received accolades for upholding coding standards in the companies I have worked with.

To illustrate these best practices, we'll consider a hypothetical project called Stack-Scraper. This project consists of a single-page website scraper that extracts questions and answers from (hypothetical) Stack Overflow. Please note that Stack-Scraper is not a functional project but serves as an example to demonstrate the ideas discussed here. The source code for this example can be found on GitHub.

I personally use VS Code as my go-to IDE, and so some of the tools and plugins mentioned here are for the same. Let's dive into the best practices and tools I employ to ensure my Python code adheres to the highest standards.

Repository Structure

A thoughtfully designed repository structure serves as the foundation for clean code development. If the structure fails to meet the project's requirements, developers may scatter code across different locations, resulting in a messy structure and decreased code re-usability. A carefully crafted repository structure plays a vital role in maintaining code organization and facilitating collaboration among developers.

It's important to note that there's no one-size-fits-all solution for repository structure. Each project may have unique requirements that warrant a specific organization scheme. However, examining a practical example can provide valuable insights. Let's consider the repository structure of the Stack-Scraper project. Here are some key points to note:

The Stack-Scraper repository employs clearly demarcated folders for different components, such as APIs (src/apis) database models (src/db_wrappers), domain-level constants (src/constants), pydantic models (src/domain_models), and scrapers (src/external_sources/scrappers) etc.. This segregation ensures a logical separation of code, enabling developers to locate and modify specific functionalities easily.
The test files in the Stack-Scraper repository follow the same hierarchy as the main repository. This practice ensures that tests remain organized and aligned with the corresponding code. Consistent test organization simplifies test management and improves code testability.

pre-commit

pre-commit is a framework that enables the execution of configurable checks or tasks on code changes prior to committing, providing a way to enforce code quality, formatting, and other project-specific requirements, thereby reducing potential issues and maintaining code consistency.

All you need to do is create a pre-commit-config.yaml file in your repository. You can install pre-commit by running following commands.

pip install pre-commit
pre-commit install

Let's examine the pre-commit-config.yaml of Stack-Scraper.

repos:
  - repo: https://github.com/ambv/black
    rev: 23.3.0
    hooks:
      - id: black
        args: [--config=./pyproject.toml]
        language_version: python3.11

  - repo: https://github.com/pycqa/flake8
    rev: 6.0.0
    hooks:
      - id: flake8
        args: [--config=./tox.ini]
        language_version: python3.11

  - repo: https://github.com/pycqa/isort
    rev: 5.12.0
    hooks:
      - id: isort
        args: ["--profile", "black", "--filter-files"]
        language_version: python3.11

  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.4.0
    hooks:
      - id: requirements-txt-fixer
        language_version: python3.11
      - id: debug-statements
      - id: detect-aws-credentials
      - id: detect-private-key

Here, we have configured 7 hooks. You can add more hooks if you need but above setup is more than enough to help you keep your code clean.

Type-hints

Python's dynamic typing allows variables to be assigned without explicit type definitions, which can lead to convenience. However, this flexibility can pose challenges in maintaining code quality, especially in larger projects where type errors and inconsistencies may occur.

Consider following example from file: src/apis/question_no.py in Stack-Scraper. Can you identify if ques_no is an integer or a string?

@stackoverflow_blueprint.route("/stackoverflow/<ques_no>", methods=["GET"])
def get_question_answer(ques_no):
    # code to do stuff

Checkout the same example now with type-hints. It clearly highlights that ques_no is expected to be an integer and get_question_answer returns a Response object.

@stackoverflow_blueprint.route("/stackoverflow/<ques_no>", methods=["GET"])
def get_question_answer(ques_no: int) -> Response:
    # code to do stuff

Doc-strings

Doc-strings are documentation strings added for each function to improve code readability. They provide additional information about the function's purpose, parameters, and return values, making it easier for developers to understand and utilize the code effectively.

You can also use doc-strings to generate automated documentation for your code using a library like pdoc. Consider the following example from Stack-Scraper and the corresponding documentation generated using pdoc library.

@stackoverflow_blueprint.route("/stackoverflow/<ques_no>", methods=["GET"])
def get_question_answer(ques_no: int) -> Response:
    """Function to fetch data for given question number

    Args:
        ques_no (int): Question number

    Returns:
        Response: Returns response object
    """
    # code to do stuff

SonarLint

SonarLint is a code analysis tool that integrates with various IDEs (as a free plugin) and helps identify and fix code quality issues, security vulnerabilities, and bugs during the development process, enabling developers to write cleaner and more reliable code. This has been a must install plugin for me over years now.

Pydantic

Pydantic is a Python library that provides runtime type checking and validation for data models. We don't aim to make Python behave as C/C++ but there are certain use cases where it's paramount to enforce type-checking. Examples: API inputs, methods in database wrapper to provide consistent in and out of data from database etc.

Stack-Scraper demonstrates an implementation for the same. By leveraging the pydantic model, StackOverflowQuestionAnswer as the output of the get_question_answer_by_ques_no method within the InMemoryDatabaseWrapper class, we establish a reliable means of retrieving questions from the database based on their unique identifiers. This design choice ensures that future changes to the database, including modifications to its schema, will not disrupt the application's functionality as long as the method's output adheres to the consistent model defined by pydantic.

Another demonstration is validating the inputs to the method upsert_question_answer in the same class as above to ensure that only allowed values go into the database.

Spell Checker

While it may initially appear counterintuitive, incorporating a spell checker into your code is a valuable practice that aligns with the other advice provided. The inclusion of a spell checker ensures code quality and readability, reinforcing the importance of attention to detail in maintaining clean and error-free code.

Firstly, it will help to avoid naming methods like upsert_qusetion_answer due to its unintuitive spelling, as it can lead to errors and confusion during usage. Notice the spelling mistake in qusetion.

Secondly, if you utilize an automatic documentation generator, it becomes essential to minimize spelling mistakes. Maintaining accurate and error-free documentation not only improves code understand-ability but also enhances the overall professionalism and credibility of the project.

Tests

Comprehensive testing is crucial for maintaining error-free and robust code, ensuring its adaptability to future changes and different developers. While aiming for 100% coverage is desirable, the focus should be on thoroughly testing the code rather than just the coverage number.

When writing test cases, it is important to consider all possible scenarios and edge cases that the code might encounter. This includes positive and negative testing, boundary testing, and error handling. An example test case for the get_question_answer function in the test/src/apis/test_question_no.py file can provide insights into designing effective test cases.

That's all for this article on maintaining clean code practices and enhancing code quality. I hope you found the insights and strategies shared here valuable. Remember, implementing these best practices and incorporating them into your development habits one step at a time can make a significant difference in your code's reliability and maintainability.

By prioritizing code cleanliness and adhering to these practices, you'll witness your code soar to new heights. Enjoy the journey of writing impeccable code and reap the rewards of robust, error-free software development. Happy coding!

Additional Tips

Here I am listing a few more generic tips/comments that you can incorporate into your coding habits.

Add comments wherever warranted in your code. If you make an assumption in the code, it's better to highlight the same in comments. Example: Line 78 in file src/external_sources/scrappers/stack_overflow.py .
Try not to define constants inside functions/classes because overtime, you could end up having bunch of these constants scattered across the code. Either define them in their respective constants file or in the respective module. Example: src/constants/api_constants.py .
Define your own Exceptions rather than using the generic one. This helps in differentiating exceptions in code to respective segregation. Example: StackOverflowException in file src/external_sources/scrappers/stack_overflow.py represents exception raised by the Stack Overflow Scrapper.
Follow OOPs for coding. Example: Inheritance by defining base class ExternalSource in file src/external_sources/external_source_base.py .
Follow REST principles for API development.
Do not add secrets into your repository. Instead use an environment file or a Secrets Management Service like AWS Secrets Manager.
Make your code environment agnostic. So, that it's easy to maintain separate development, testing, staging and/or production environments. Example: src/config.py .
Peg versions of the libraries that you install in requirements.txt file. It helps in avoiding application failure because of breaking upgrades to libraries.

How to develop a Wox Plugin using Python

Ashutosh Kumar — Sat, 15 Jul 2023 11:49:01 +0000

Read this article on Medium published by gitconnected.

Wox is a full-featured launcher for the Windows platform. It’s akin to Mac’s Spotlight and can be augmented by a plethora of plugins.

I have been using Wox (with the almighty Everything plugin) for several years now and have loved it. But I have always resented the unavailability of a working English dictionary plugin that would also work offline. So, I embarked on developing a plugin for myself but was frustrated by the lack of documentation available to do the same.

Below, I am sharing how I developed a dictionary plugin, Easy Dictionary, using Python. I hope it would help you in your quest to develop another Python-based plugin. Easy Dictionary can be installed from the official Wox plugin website. The source code is available on GitHub. Please check it out and you’d understand why I named it an “Easy” dictionary.

Alright, all we need to do is follow below simple steps.

We need to create a python file to contain our code. Let’s call it main.py (file name could be anything). Add a class (class name could be anything) inheriting from wox.Wox and instantiate the same as follows:

# Only for type-hinting in Python
# For more info: https://docs.python.org/3/library/typing.html
from typing import Dict, List

# No need to worry about the presence of the `Wox` class. Wox runtime has its own 
# python environment that contains the required `Wox` class
from wox import Wox


# Class name could be anything
# NOTE: Don't forget to inherit from `Wox`
class EDict(Wox):
    # Overriding the `__init__` is NOT mandatory
    def __init__(self, *args, **kwargs) -> None:
        """Initializer for `EDict` class"""
        # Add all the custom initialization required for the `EDict` class

        # NOTE: If overriding the `__init__` method, make sure to call the
        # parent class initializer like below
        super().__init__(*args, **kwargs)

    # NOTE: We must override the `query` method
    # `query` method is called when a user queries the plugin using its trigger
    # keyword in Wox. Wox passes the user input as an argument
    def query(self, key: str) -> List[Dict[str, str]]:
        results: List[Dict[str, str]] = []
        # NOTE: Each item on the list must be a dictionary that should contain 
        # at least following keys. For more info, check out the annotated screenshot below
            # "Title": Main search result to show in Wox
            # "IcoPath": Path of image/icon for the search result in Wox
            # "SubTitle": Subtitle of the search result in Wox

        # Our core logic goes here or in any other method that we can call here

        return results


if __name__ == "__main__":
    # NOTE: Don't forget to instantiate the above class
    # No need to return anything
    EDict()

We need to create another file called plugin.json (the name must be the same). Wox looks for this JSON file to get details about the plugin and instantiates the same at runtime. Below plugin.json file is for the Easy Dictionary plugin.

The plugin.json file requires the following details:

ID: This must be a UUID that we generate online or using any programming language. It shouldn’t clash with the ID of any other plugins available for Wox.
ActionKeyword: It is the trigger keyword for the plugin when using Wox. See the below screenshot showing Easy Dictionary getting triggered on typing “ed”.

Name, Description, Author, Version: These are self-explanatory.
Language: Wox supports writing plugins in many languages as the communication is based on RPC. So, language is not a barrier for Wox plugins. Here, we need to specify the language in which we are writing the plugin (“python” in this case).
Website: This can be empty string or we can link to a website for our plugin.
IcoPath: Path of image/icon we wish to use for our plugin. This image/icon must be part of the plugin package (we will be packaging the plugin shortly).
ExecuteFileName: It specified the entry point for our plugin. Note that it’s the same main.py file that we created earlier.

{
    "ID":"0c6569f79d49409e8e5e42e1ec8bb035",
    "ActionKeyword":"ed",
    "Name":"Easy Dictionary",
    "Description":"Provides an offline English Dictionary",
    "Author":"ashutosh",
    "Version":"2.2.0",
    "Language":"python",
    "Website":"https://github.com/ashu-tosh-kumar/Wox.Plugin.eDict",
    "IcoPath": "icons\\edict.png",
    "ExecuteFileName":"main.py"
  }

We are almost done. Next we need to just grab the files main.py , plugin.json, folder named icons with files edict.png (used in plugin.json) & edit.ico (used in main.py) and zip them together into Wox.Plugin.<plugin_name>.zip. In our case, it’s Wox.Plugin.eDict.zip.

Once zipped, change the zip format manually to .wox. So, finally we would have the zipped file: Wox.Plugin.eDict.wox.

The file Wox.Plugin.eDict.wox can be dragged and dropped onto Wox launcher to install it manually. Or, we can share it on Wox’s official website for wider use by creating an account and uploading the above file.

That’s it. Follow the above steps and you can create your own Wox plugins using our old chap Python language. For the sake of completeness, let’s go through the actual code of Easy Dictionary below.

First, let’s have a look at the files that we would need to package for the Easy Dictionary. Please checkout the source code on GitHub.

|-icons  # Folder
| |-edict.ico
| |-edict.png
|-dictionary_compact_with_words.zip
|-main.py
|-plugin.json
|-spell.py

icons: icons folder contains the image edict.png and icon edict.ico used in the project.
dictionary_compact_with_words.zip: It contains the unabridged Webster dictionary in json format. We have zipped it to reduce the plugin size. We unzip it for use in Python in __init__ method of class EDict in main.py

class EDict(Wox):
    """Easy Dictionary Class used by Wox"""

    def __init__(self, *args, **kwargs) -> None:
        """Initializer for `EDict` class"""
        # Unzipping the dictionary for usage
        with ZipFile(DICTIONARY_ZIP_FILE, "r") as zip_file:
            with zip_file.open(DICTIONARY_JSON_FILE) as edict_file:
                self._edict = load(edict_file)

        # Key "cb2b20da-9168-4e8e-8e8f-9b54e7d42214" gives a list of all words
        # For more info, checkout: https://github.com/matthewreagan/WebstersEnglishDictionary
        words = self._edict["cb2b20da-9168-4e8e-8e8f-9b54e7d42214"]
        self._spell_correct = SpellCorrect(words)

        super().__init__(*args, **kwargs)

main.py: This is our starting point for our plugin. We override the query method in class EDict to provide results to user query. We use the method _format_result to format each result as a dictionary as explained earlier. Some formatting rules in _format_result are in place because of the underlying dictionary used. Moreover, if we fail to find a definition for user input key, we try to auto-correct the word and show results for the auto-corrected word instead.

def query(self, key: str) -> List[Dict[str, str]]:
    """Overrides Wox query function to capture user input

    Args:
        key (str): User search input

    Returns:
        List[Dict[str, str]]: Returns list of results where each result is a dictionary
    """
    results: List[Dict[str, str]] = []
    key = key.strip().lower()

    if not key:
        # Avoid looking for empty key
        return results

    try:
        # Look for the given key
        definitions = self._edict[key]

        # Format results in the form of a dictionary
        results = self._format_result(definitions, key, MAXIMUM_RESULTS, False)
    except KeyError:
        # Try correcting the key and looking again with the corrected key
        # This is an additional feature where we try to auto-correct the user input
        # This helps in case of spelling mistakes
        try:
            corrected_key = self._spell_correct.correction(key)
            definitions = self._edict[corrected_key]
            results = self._format_result(definitions, corrected_key, MAXIMUM_RESULTS, True)
        except KeyError:
            # Word doesn't exist in our dictionary
            pass

    return results

plugin.json: Explained earlier.
spell.py: A simple implementation of probability based method to auto-correct a word. Implementation taken from norvig.com. This file is used by main.py.

Streamline Your TODOs with TODO Notifier for Python projects

Ashutosh Kumar — Fri, 21 Apr 2023 05:12:57 +0000

Full article on Medium.

Are you tired of tracking your TODOs in Python projects? Do you wish there was a tool that could scrape and summarize all your TODOs for you? Checkout TODO Notifier, an open source project available on GitHub and PyPi.

At my last company, I had the pleasure of using a tool that summarized all the TODO items in a Python project. It would send periodic reminders to individual developers to complete them. The tool was helpful and could identify expired and upcoming TODOs with an expected completion date.

When I moved to a new company, I realized how much I missed that tool. So, I decided to take matters into my own hands and develop a similar tool myself. It was a challenging but rewarding experience. And I’m excited to share my journey of building a Python TODO management tool.

With TODO Notifier, you can set up automated summaries of TODO items in your code by module. It also lists all expired and upcoming TODO items based on expiry date. Moreover, you can customize the tool to generate and send these summaries over email. And send reminders to individual developers (upcoming feature) to finish their TODO items.

Let’s first see the TODO Notifier in action. I will be using TODO Notifier on itself to generate the default summaries. Please note that I am not using any notifier for this demo.

Setting up TODO Notifier is simple. Follow the detailed article on Medium and you will have TODO Notifier ready in no time. If you want more information, please check out the GitHub or PyPi page.

TODO Notifier is flexible and customizable. It’s open source and free to use. You can adjust the date format to match your project’s needs. You can also send the generated summaries over Email or any other notification service. You can exclude certain modules or files from the summaries.

TODO Notifier is a powerful tool for Python developers who want to streamline their TODO management. With its easy setup and customizable options, it’s a must-have for any project with many developers and lots of TODOs to track.

If you’re interested in trying out TODO Notifier, you can find the code on GitHub and install it with pip. Give it a try and let me know what you think!

How to start with competitive coding from ZERO

Ashutosh Kumar — Fri, 06 Dec 2019 19:38:07 +0000

How to Start with Competitive Coding from Zero?

-Photo by Kobu Agency on Unsplash

This is a question that has haunted me for years, and I could never begin for a long time. Finally, it turned out to be a simple lesson: “Start by doing”, and that’s all that is required. The only problem in this statement is what exactly to start with.

I am from a non-CSE Department and chose to have a career in Software Engineering. So, I had to start from zero myself. After 8 months of coding practice, I landed an offer from a good MNC. Here I am laying down a simple step-wise plan that might help you to start from zero. Though it is not the only plan that would work and you don’t need to stick to it. But it helps to have a concrete plan to compare with in case you have your own road map. In fact, once you start having a hang of coding (after solving 60–80 easy-level questions), you would not even need to further follow this plan. But, it would still be handy to check if you miss any topic.

This post has six sections. The first section includes basic data structures and algorithms that you need to learn. Completing Section 1 might be helpful in coding tests of some companies. Section 2 includes Dynamic Programming and Greedy Algorithms. Completing Sections 1 and 2 will be good enough for fresher coding tests of most companies. Section 3 includes graphs that you encounter in coding tests of a few companies. Section 4 contains some other specific topics/data structures/algorithms that I have encountered in coding tests of a few companies. But, you won’t find questions too often on these topics but it’s better to learn them too if you could get enough time). Section 5 includes some extra topics/algorithms that are generally taught in Algorithms courses but are not important for placement tests. Section 6 includes FAQs about placements that I have encountered people asking in different forums.

It would be better if you approach and master these sections sequentially (except Section 4; you can do it with Section 3). But, you can have your own preference depending on what works best for you.

Note : Please note that I keep editing this article to best add all the relevant information/resources. I maintain a history of all the edits that I make so that it is easy for you to follow the changes. So, checkout the Edits Section at the end of this post if anything is added after you last visited the article.

Section 1

Let’s first have a look at the key things required for competitive coding:

Programming Language
Data Structures
Algorithms
Online Coding Practice

A. Programming Language

Edited Image from towardsdatascience

The first step is to choose a programming language (only choosing and not mastering). You only need to know some basic functionalities and shouldn’t bother about mastering it. Many people (including me) fall into the trap of trying to master the language before coding.

You could never learn a language without starting to use it.

So, to learn the required basic functionalities, head to HackerRankand log in/create your account. After login look for “Language Proficiency” Modules as shown in following image:

Language Proficiency Module

Choose a programming language and solve the first 15–20 questions. That’s it! Only the first 15–20 questions and you are all set with all the basic functionalities required. Don’t bother about the choice of the language at this point. If you are good at coding with one programming language, you can pick up any other. I coded with Python for around 7 months and then picked up C++ in 2–3 days. I first read some of its theory (see Resources at the end of this post) and then practised with it for next 3–5 days.

Note: If you don’t have a strict personal preference or are not sure about the programming language, here’s a point that may help you. I only know Python and C++ and so my point here is only about these two languages. If you are an absolute beginner, you will find it easier to start with Python than C++. But, you have to learn C++ because sometimes (only 1–2 companies like JP Morgan Quant Profile) don’t allow Python (I don’t know why). And, many companies ask questions about the output of a given C/C++ code in the form of MCQs and OOP concepts in C++. So, if you have around one semester (3–4 months) then Python may be your best bet as the learning will be fast with Python. But, if you have at least two semesters (8 months or more) then you can either choose C++ or Python first and then learn C++ later on (that’s what I did). But, learning will be slow initially with C++ but normal once you have enough practice.

B. Data Structures

Photo by Harsh Goel on HackerEarth

Data structures are the basic building blocks of competitive coding. It is better to first read about the basic data structures mentioned below.

Array
Linked List
Stack
Queue
Strings
Hashing* (Only usage, you can read theory later on)
Binary Trees**
Binary Search Trees**
Heap**

You can learn about the above data structures from GeeksforGeeks. Read their theory and learn how to implement them in the programming language of your choice. GeeksforGeeks contains the implementation of these data structures in different programming languages.

Once you master and have some experience with these basic data structures (№1–6), you can go on to learn the complex ones mentioned at the end of this post.

*Note: I have mentioned hashing here because you may find it helpful in a lot of problems. You need not know its theory at this point. Most of the programming languages offer built-in implementation of hashing. But its theory is very important for coding interviews. You should check out its theory from the book Introduction to Algorithms mentioned in the Resources section.

**Note: I am not counting binary trees, binary search trees and heaps in basic data structures. And, you need not to finish them right now to move on to the next section. I have mentioned them here because there are lectures in the next section on these topics. So, complete only up to Hashing and move on to the next section. After completing lectures on trees and heaps in the next Section, you can learn their implementation from the above links.

C. Algorithms

Edited Image from Coursera

Algorithms are the most important part of competitive programming. If you master algorithms, you are more than 50% done.

There is no substitute for a proper course on Algorithms. It would be better if you both watch a lecture and read from GeeksforGeeks. For example, the Algorithms course ( CS21003 ) offered by the CSE Department in my college (IIT Kharagpur) is good. I would recommend that you take that course (if possible) as a “depth/breadth” and not as an “additional” (for obvious reasons). CS21003 is offered in even semesters for non-CSE students. You can get the course if you have an 8+ CGPA. I am not sure about the below 8 CGPA because the cutoff depends upon all the students who have applied.

If you are from some other college or can’t take that course for some reason, I am adding relevant online lectures below. Feel free to skip these lectures if you are already taking any course in Algorithms.

I am not aware of many online courses on Algorithms except those provided by MIT OCW which you can find on their website or YouTube. MIT has many courses on Algorithms, and you may find it overwhelming to complete them. Moreover, it’s not required to complete them for fresher coding tests. So I am mentioning selective lectures from the courses MIT 6.006 Introduction to Algorithms, Fall 2011 and MIT 6.046J Design and Analysis of Algorithms, Spring 2015 that you can watch in the sequence mentioned below:

[MIT 6.006] R1. Asymptotic Complexity, Peak Finding
[MIT 6.006] R5. Recursion Trees, Binary Search Trees (only Recursion trees part)
[MIT 6.006] Lec1: Algorithmic Thinking, Peak Finding
[MIT 6.006] Lec2: Models of Computation, Document Distance
Read about Recursive Thinking from GeeksforGeeks
[MIT 6.006] Lec3: Insertion Sort, Merge Sort
[MIT 6.006] Lec4: Heaps and Heap Sort
[MIT 6.006] Lec5: Binary Search Trees, BST Sort
[MIT 6.006] Lec7: Counting Sort, Radix Sort, Lower Bounds for Sorting
[MIT 6.046J] Lec2. Divide & Conquer: Convex Hull, Median Finding
[MIT 6.046J] R1. Matrix Multiplication and the Master Theorem (matrix multiplication is optional, but do check master theorem)
[MIT 6.046J] R4. Randomized Select and Randomized Quicksort

The above lectures should be enough to start coding online.

While you are watching lectures, also start practising coding online. That is, suppose you finished a lecture on merge sort then you should try to code it yourself. You should always know to code the routines of all major sorting algorithms by yourself. For example, many companies have asked Array Inversion. You will need to write a modified routine of Merge Sort to get an O(nlogn) complexity. At this point, you already know a language and have completed basic data structures. So, you should also start doing questions about those data structures as explained in the next section.

D. Online Coding

Photo by Clément H on Unsplash

Practice is the ultimate key to mastering anything you want to learn.

There are several websites to practice coding like LeetCode, HackerRank, InterviewBit, Codeforces, and CodeChef. I have used only the first three. Start with LeetCode if you have at least two semesters (8 months) to prepare else start with InterviewBit.

So far, you have completed Data Structures and started Algorithms. So, head to the LeetCode Problems and filter out the questions based on topics that you have already covered. Then, sort the questions based on their level of difficulty. You should start with the Easy level questions. Easy doesn’t mean “easy” and you may not be able to solve them on your first try. Solve 10–15 questions on each of the topics that you have covered so far and then head to the next section.

Having done 10–15 Easy level questions on all the topics, you should start doing Medium level Questions on them. I would recommend that you should start doing Medium level questions once you start Section 2. Once you start doing Medium level questions, you would feel easy-level questions to be “easy”.

NOTE: InterviewBit has a programming practice track that you should complete for your coding tests. It includes questions that have already been asked in coding tests of different companies. And, you will find repetitions of many of these questions in your coding tests. So if you start with any other website like LeetCode , make sure that you also complete this track. It may take around 1–2 months to complete this track, so plan accordingly. If possible, also complete the Top 100 liked questions of LeetCode after completing the InterviewBit practice track as you may find many repetitions from there also. Don’t worry, you would have already solved most of these 100 questions during your practice.

Section 2

If you have come this far, then you would have gained very good coding skills. There are two important algorithmic paradigms that are important to master and I didn’t mention them in the above Algorithms section. These are Dynamic Programming and Greedy Algorithms. You can watch the following lectures for the same:

[MIT 6.006] Lec19: Dynamic Programming I: Fibonacci, Shortest Paths
[MIT 6.006] Lec20: Dynamic Programming II: Text Justification, Blackjack
[MIT 6.006] Lec21. DP III: Parenthesization, Edit Distance, Knapsack
[MIT 6.046J] R6. Greedy Algorithms (Also read about Greedy algorithms from GeeksforGeeks via example problems shown there. You need to practice a bit to develop an intuition on where you can use the Greedy approach.)

The path is like as above for these two topics. Watch the video lectures and then solve 10–15 questions on each topic. Dynamic Programming and Greedy approaches are important. Practice as many questions as you can on them.

Section 3

Image from Annalect

So far, you have covered almost everything required to be a good competitive programmer except graphs. If you still got time, you can go on to learn graphs*. You may find the following resources useful:

Read about Graphs as a data structure on GeeksforGeeks
Read about representations of Graphs from GeeksforGeeks
[MIT 6.006] Lec13: Breadth-First Search (BFS)
[MIT 6.006] Lec14: Depth-First Search (DFS), Topological Sort
[MIT 6.046J] Lec12: Greedy Algorithms: Minimum Spanning Tree (This lecture includes Prim’s MST Algorithm and Kruskal’s MST Algorithm**)
[MIT 6.006] Lec15: Single-Source Shortest Paths Problem (SSSP Problem)
[MIT 6.006] Lec16: Dijkstra (for SSSP Problem)
[MIT 6.006] Lec17: Bellman-Ford(for SSSP Problem)
[MIT 6.046J] Lec.11: Dynamic Programming: All-Pairs Shortest Paths (ONLY Floyd Warshal Algorithm)

With graphs, you have covered almost everything required assuming that you were also practising online. There are some specific topics left that I have encountered sometimes in coding tests of a few companies. You can find these topics in the next section.

*NOTE : If you are overwhelmed with so much by now or don’t have much time for Graphs then finish only up to BFS (№3) and DFS (№4). BFS and DFS will be enough for 95–99% of the questions that you may encounter on Graphs in coding tests. Only 2–3 core CSE companies open for non-CSE students ask very difficult questions on graphs like Nutanix. So, don’t bother much about graphs for now. Additionally, you can learn Topological Sort.

**NOTE : For implementing Kruskal’s Minimum Spanning Tree Algorithm, you would need to learn Disjoint Data Set Structure. It’s mentioned in Section 4 and is a very helpful data structure for a lot of problems. You can read about it first from here and then here.

Section 4

I have encountered the following topics sometimes in coding tests of a few companies. You can read about them if you got enough time.

Bits Manipulation : Bits manipulation is an important topic for coding as well as interviews. It can save you a lot of time and complexity in certain questions. Walmart asked questions only on Bits Manipulation this time in both the Coding and Data Science profiles. You can read about Bits Manipulation from HackerEarth Tutorial.
Dequeue : I have found usage of Dequeues in only one specific question asked in the coding test of two companies. Both Python and C++ offer built-in implementation for Dequeues.
Disjoint Set Union (DSU) Data Structure: DSU is important and used in Kruskal’s Minimum Spanning Tree Algorithm and in a bunch of other problems. I have encountered questions on DSU in a few coding tests including that of AB InBev.
Trie Data Structure : Tries can come in handy in specific sets of problems on strings and bits. I was asked to implement a “set” for strings with O(n) or O(logn) complexity in my coding interview. You can do the same using Tries.
Concepts of Object Oriented Programming (OOP): You should understand the OOP concepts like Encapsulation and Interface. Questions on OOPs in general and OOPs in C++ are often asked in coding tests in the form of MCQs.
Regular Expression (regex): I encountered one question on regex in my entire placement (Trexquant). So it’s not important but if you have time, you may give it a shot in case you get unlucky.

Section 5 (not important for Placement tests)

I am adding this section to include height-balanced binary trees and string-matching algorithms. Because they are generally taught in algorithms courses.

But, they are not important for placement tests for all those companies that are open to non-CSE students. Most of the programming languages offer built-in implementation of pattern matching for strings. And, questions on height-balanced binary trees are generally time-consuming.

[MIT 6.006] Lec.6: AVL Trees, AVL Sort (height-balanced binary trees)
[MIT 6.046J / 18.410J Fall 2005] Lec.10: Red-black Trees, Rotations, Insertions, Deletions (height-balanced binary trees)
[MIT 6.006] Lec.9: Table Doubling, Karp-Rabin (only Rabin-Karp part) (for pattern matching in strings)
You can read about KMP Algorithm from GeeksforGeeks (for pattern matching in strings)

Section 6 (FAQs)

Q1. Is STL allowed in C++ in placement tests?

Yes, you can import all the basic libraries using #include . But, if the question asks to implement certain functionality already provided by a built-in function, then the placement organizer may block the usage of that particular function.

Q2. Can libraries be imported in Python in placement tests?

Yes, you can import any library you want. But the organizer can limit your options in questions on Data Analysis like in the Publicis Sapient Data Science profile. As a rule of thumb, put the usage of libraries to a least.

Q3. Is there any downside to using Python given that Python takes more time than C/C++?

No, the time limits to solutions are language specific. In general, only the complexity like O(logn) affects the acceptance of your code. But, note that this time Walmart put an absolute time limit on questions, likely a mistake from their side. An O(nlogn) solution didn’t get accepted in C++ or Python but an O(n²) solution got accepted in C. This was a rare incident and there isn’t anything that you can do in such situations.

Q4. Are we asked to solve programming questions in placement Interviews?

It depends on the interviewer. Some companies like Amazon asks to write proper code, (not pseudo-code) on paper. You need to practice writing code on paper beforehand. For other companies, the interviewer can either ask you to write proper code or pseudo code or may not ask a programming question at all. In such cases, your interview may revolve around your resume and other Aptitude questions/Puzzles.

Q5. Is OOPs important for placements?

OOPs is not important for coding tests and you won’t need to write classes or use inheritance etc. But, OOPs is important for MCQs in coding tests. Some companies have a joint MCQ & Coding test like Amazon. Also, OOPs is very important for interviews.

Q6. Is it important to have software internships/experience to bag a placement in software engineering?

No, except for the walk-in interviews and cases when companies go for a resume-based shortlist (generally happens after Day 2). The only thing that is important is your coding test. I didn’t have any software experience myself and only had an ML internship. So my interview went around my ML project. In your case, it could revolve around your resume. Even so, having an experience in software is always a plus.

Q7. How much CGPA and CV matter for placements?

There’s no straightforward answer to this question. I would try to answer this question from the perspective of the timeline of events in a job process for non-core software/ML profiles. First of all, companies open up and they put a general or department-wise cutoff. 7+ in most cases but higher in a few cases like Wells Fargo put 8.5+ for Mechanical Engineering. That’s where CGPA matters. After that, you give your coding tests and companies shortlist you based on your coding test performance. There may be 1–2 exceptions where the company also take into consideration your resume. So CV might play a role in 1–2 cases. After that, you go for your interviews and in most cases, only your performance affects your selection. There could be a few exceptions depending on factors like interviewer, profile and fellow candidates. So, your coding skills and performance in interviews (both technical and HR) are most important. Other things in general only help in clearing cutoffs. Even so, a good CGPA and CV are always a plus.

That’s it. If you have come this far, you should have gained excellent coding skills. All the best for your placements/coding challenges.

I have covered pretty much everything. I will update this post if I find anything missing. Thank You for reading and let me know in the comments about your experience in learning coding. If you liked reading this article, give me some claps.

Resources

Some other resources that you may find helpful:

GeeksforGeeks: A wonderful website for all the theories and explanations of different solutions to various programming questions. GeeksforGeeks would be your Handbook for coding.
[BOOK] Introduction to Algorithms by Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, Clifford Stein: For theory and a better understanding of any specific topic in Algorithms.
Stack Overflow: For doubts or clarifications in programming questions/algorithms/your code.
Lecture Notes of MIT Introduction to C++: In case you choose to learn C++, lecture notes of this course might be helpful. It’s not necessary to complete the whole course, only read the lecture notes.
CS21003 Course by Prof. Abhijit Das, CSE Dept. IIT Kharagpur: For checking the proper syllabus of this course.
Lectures Notes for CS21003 Course: Credits: Anubhav Jain, Student CSE Dept. IIT Kharagpur

Edits

Someday

Corrected some linguistic mistakes and added more clarifications at certain points.
Added Top 100 liked questions of LeetCode in the last Note of Section 1.

12-Feb-2020

Corrected some linguistic mistakes and made some points clearer.
Moved the OOP section from Resources to Section 4.
Added GeeksforGeeks in the Resources section.
Added some points in the section of ‘Programming Language’ to help choose between Python and C++.

10-March-2020

Added a few images to balance the wordiness of the post.

17-May-2020

Added resource for theory of Hashing in Data Structures in Section 1 (first Note there).
Added 3 lectures (№2:Recursion Trees, №5:Recursive Thinking, №11:Matrix Multiplication and Master Theorem) in Algorithms of section 1.
Added 1 lecture for Greedy Algorithms (№4) in Section 2.
Added 3 lectures (№5:Minimum Spanning Tree, №8:Bellman Ford, №9:Floyd Warshal Algorithm) in Section 3.
Added Section 5 for certain topics/algorithms that are generally taught in algorithms courses but are NOT important for placement tests.
Added CS21003 website in the Resources section.
Added CS21003 course notes of Anubhav Jain, student CSE Dept. IIT Kharagpur.

18-May-2020

Shifted the Edits section after the Resources section.
Changed some images and a bit of face-lift. No change in content.
Added Regular Expression (regex) in Section 4 (Point №6).

22-May-2020

Added missing link to lecture on Bellman-Ford (№8) in Section 3

02-July-2020

Corrected some linguistic mistakes.
Added some explanation at the start of Section 4.

23-July-2020

Added Section 6 for FAQs (added Q.1, Q.2, Q.3) about placement tests and placement Interviews that I often encounter on various forums. I will keep adding relevant questions here.
Added the following Note before Section 1: “ Note : Please note that I keep editing this article to best add all the relevant information/resources. I maintain a history of all the edits that I make so that it is easy for you to follow the changes if you are revisiting this article. So in such a case, please check out the Edits Section at the end of this post to check if anything is added after you last visited the article.”

24-July-2020

Added Q.4 in Section 6.
Added GeeksforGeeks link to Floyd Warshall Algorithm (Lecture 9) in Section 3.
Edited solution of Q.2 in Section 6 to include libraries (heapq, collections, random and bisect) that are generally imported in Python for CP problems.

10-Sep-2020

Italicized all the notes to differentiate them from the text. No change in content.
Moved the note in Section 1: A) Programming Language from middle to end. No change in Content.
Stripped down answers to Q.2 and Q.3 in Section 6 : FAQs

08-Oct-2020

Added Q.5, Q.6 and Q.7 in Section 6: FAQs

25–July-2023

Improved the writing w.r.t. Hemingway Editor. Re-worded the article, with no material changes.