Forem: Ashutosh Sarangi

(aiohttp & asyncio) vs Requests: Comparing Python HTTP Libraries

Ashutosh Sarangi — Tue, 04 Nov 2025 06:34:22 +0000

1. Requests - The Simple Synchronous Library

What it is:

Synchronous blocking HTTP library
Simple, intuitive API
Most popular for basic HTTP operations
Blocks execution until response is received

When to use:

Simple scripts
Sequential API calls
Learning/prototyping
When performance isn't critical

Example:

import requests
import time

# Basic GET request
response = requests.get('https://api.github.com/users/github')
print(response.json())

# Making multiple requests (BLOCKING - one at a time)
def fetch_multiple_sync():
    urls = [
        'https://api.github.com/users/github',
        'https://api.github.com/users/google',
        'https://api.github.com/users/microsoft'
    ]

    start = time.time()
    results = []

    for url in urls:
        response = requests.get(url)
        results.append(response.json())

    print(f"Time taken: {time.time() - start:.2f} seconds")
    # Output: ~3 seconds (1 second per request, sequential)
    return results

# POST request with headers and data
response = requests.post(
    'https://httpbin.org/post',
    json={'key': 'value'},
    headers={'Authorization': 'Bearer token'},
    timeout=5
)

# Session for connection pooling
session = requests.Session()
session.headers.update({'User-Agent': 'MyApp'})
response = session.get('https://api.github.com/users/github')

Pros:

✅ Dead simple API
✅ Excellent documentation
✅ Perfect for beginners
✅ Built-in JSON decoding
✅ Session management

Cons:

❌ Synchronous/blocking
❌ Slow for multiple requests
❌ Not suitable for high-concurrency

2. asyncio - The Foundation (Not an HTTP Library!)

What it is:

Asynchronous I/O framework built into Python 3.4+
NOT an HTTP library itself
Provides the foundation for async programming
Event loop for concurrent operations

Key Concepts:

async def - Defines coroutine functions
await - Waits for async operations
asyncio.gather() - Runs multiple coroutines concurrently

Example (without HTTP):

import asyncio
import time

# Basic async function
async def say_hello(name, delay):
    await asyncio.sleep(delay)  # Non-blocking sleep
    print(f"Hello, {name}!")
    return name

# Running async functions
async def main():
    start = time.time()

    # Sequential execution (3 seconds total)
    await say_hello("Alice", 1)
    await say_hello("Bob", 1)
    await say_hello("Charlie", 1)

    print(f"Sequential time: {time.time() - start:.2f}s")

    # Concurrent execution (1 second total!)
    start = time.time()
    await asyncio.gather(
        say_hello("Alice", 1),
        say_hello("Bob", 1),
        say_hello("Charlie", 1)
    )

    print(f"Concurrent time: {time.time() - start:.2f}s")

# Run the async function
asyncio.run(main())

Why asyncio alone isn't enough for HTTP:

# This WON'T work - requests is synchronous!
import requests
import asyncio

async def fetch_url(url):
    # ❌ requests.get() blocks the event loop!
    response = requests.get(url)  
    return response.json()

# Even with asyncio.gather(), this is still sequential
# because requests.get() blocks

3. aiohttp - Async HTTP Client/Server

What it is:

Asynchronous HTTP client and server
Built on top of asyncio
Non-blocking I/O for concurrent requests
Supports WebSockets

When to use:

Multiple concurrent HTTP requests
High-performance web scraping
API integrations with many endpoints
WebSocket connections
Building async web servers

📊 Side-by-Side Comparison---

🔥 Real Performance Example

Let me show you a real-world scenario comparing both:

import requests
import aiohttp
import asyncio
import time

# Scenario: Fetch data from 20 different API endpoints

urls = [f'https://jsonplaceholder.typicode.com/posts/{i}' for i in range(1, 21)]

# ============ REQUESTS (SYNCHRONOUS) ============
def fetch_with_requests():
    start = time.time()
    results = []

    for url in urls:
        response = requests.get(url)
        results.append(response.json())

    elapsed = time.time() - start
    print(f"Requests (sync): {elapsed:.2f} seconds")
    return results

# ============ AIOHTTP (ASYNCHRONOUS) ============
async def fetch_with_aiohttp():
    start = time.time()

    async with aiohttp.ClientSession() as session:
        async def fetch_one(url):
            async with session.get(url) as response:
                return await response.json()

        tasks = [fetch_one(url) for url in urls]
        results = await asyncio.gather(*tasks)

    elapsed = time.time() - start
    print(f"aiohttp (async): {elapsed:.2f} seconds")
    return results

# Run both
fetch_with_requests()  # ~5-10 seconds
asyncio.run(fetch_with_aiohttp())  # ~0.5-1 second

Results:

Requests: 8.5 seconds (sequential)
aiohttp: 0.9 seconds (concurrent)
Speedup: ~9.4x faster! 🚀

🔍 What exactly happens with `session.get(url)`?

tasks = [fetch_one(url) for url in urls]

Answer: NO, the API is NOT called yet!

Here's what's actually happening step by step:

Step 1: Understanding what `fetch_one(url)` returns

async def fetch_one(url):
    async with session.get(url) as response:
        return await response.json()

# When you call this:
task = fetch_one(url)  # ❌ API NOT called yet!

When you call an async def function without await, it returns a coroutine object (not the result). The function body doesn't execute yet!

import asyncio

async def fetch_one(url):
    print(f"Fetching {url}")  # This won't print yet!
    return "data"

# This creates a coroutine object but doesn't run it
task = fetch_one("https://example.com")
print(type(task))  # <class 'coroutine'>

# The function body hasn't run yet!
# "Fetching https://example.com" hasn't printed!

# To actually run it, you need await:
result = await task  # NOW it runs and prints "Fetching..."

Step 2: What's in the tasks list?

tasks = [fetch_one(url) for url in urls]
# tasks = [<coroutine>, <coroutine>, <coroutine>, ...]
# These are "suspended" function calls waiting to be executed

The tasks list contains coroutine objects - basically "promises" or "blueprints" of work to be done, but the work hasn't started yet.

Step 3: What does `asyncio.gather()` do?

results = await asyncio.gather(*tasks)

NOW the magic happens! asyncio.gather():

Takes all the coroutine objects
Schedules them to run concurrently on the event loop
Waits for all of them to complete
Returns their results as a list

`*` and `**` Unpacking

`*` (Single asterisk) - Unpacking Iterables

# Unpacks lists, tuples, sets, etc.
tasks = [task1, task2, task3]

# These are equivalent:
asyncio.gather(*tasks)
asyncio.gather(task1, task2, task3)

# Another example:
numbers = [1, 2, 3]
print(*numbers)  # Same as: print(1, 2, 3)
# Output: 1 2 3

It's not just for lists - it works with ANY iterable (lists, tuples, sets, generators, etc.)

`**` (Double asterisk) - Unpacking Dictionaries

# Unpacks dictionaries as keyword arguments
config = {'timeout': 10, 'headers': {'User-Agent': 'MyApp'}}

# These are equivalent:
session.get(url, **config)
session.get(url, timeout=10, headers={'User-Agent': 'MyApp'})

# Another example:
def greet(name, age):
    print(f"{name} is {age} years old")

person = {'name': 'Alice', 'age': 30}
greet(**person)  # Same as: greet(name='Alice', age=30)

Python Performance Optimization: Detailed Guide

Ashutosh Sarangi — Fri, 03 Oct 2025 07:06:32 +0000

**product Object.model_dump()

Python Performance Optimization: Detailed Guide

1. Overview: Optimize What Needs Optimizing

Why This Matters

Premature optimization is the root of all evil. Optimizing the wrong parts of your code wastes time and can make code harder to maintain.

The Right Approach

# Step 1: Get it right first
def calculate_total(items):
    return sum(item['price'] * item['quantity'] for item in items)

# Step 2: Test it's right
def test_calculate_total():
    items = [{'price': 10, 'quantity': 2}, {'price': 5, 'quantity': 3}]
    assert calculate_total(items) == 35

# Step 3: Profile if slow
import cProfile
cProfile.run('calculate_total(large_item_list)')

# Step 4: Optimize based on profiling results
# Step 5: Repeat testing after optimization

Key Point: Always profile first. What you think is slow might not be the bottleneck!

2. Sorting Optimization

❌ Avoid: Using Comparison Functions

# BAD - Comparison function called O(n log n) times
def compare_by_age(person1, person2):
    if person1['age'] < person2['age']:
        return -1
    elif person1['age'] > person2['age']:
        return 1
    return 0

people = [{'name': 'Alice', 'age': 30}, {'name': 'Bob', 'age': 25}]
# Python 2 style - SLOW
# people.sort(cmp=compare_by_age)

Why This is Slow:

Comparison function is a Python function call (expensive!)
Called O(n log n) times during sorting
For 10,000 items, ~130,000 function calls
Each call has Python interpreter overhead

✅ Use: key Parameter with operator.itemgetter

from operator import itemgetter

# GOOD - key function called only O(n) times
people = [
    {'name': 'Alice', 'age': 30, 'salary': 100000},
    {'name': 'Bob', 'age': 25, 'salary': 80000},
    {'name': 'Charlie', 'age': 35, 'salary': 120000}
]

# Sort by single field
people.sort(key=itemgetter('age'))
# [{'name': 'Bob', 'age': 25, ...}, {'name': 'Alice', 'age': 30, ...}, ...]

# Sort by multiple fields (age, then salary)
people.sort(key=itemgetter('age', 'salary'))

# For tuples/lists, use index
data = [(1, 'apple', 5), (2, 'banana', 3), (3, 'cherry', 8)]
data.sort(key=itemgetter(2))  # Sort by third element
# [(2, 'banana', 3), (1, 'apple', 5), (3, 'cherry', 8)]

Why This is Fast:

itemgetter is implemented in C
Key function called only once per item (O(n) calls)
Native comparisons on extracted keys (fast C code)
For 10,000 items: 10,000 key calls vs 130,000 comparison calls

✅ Use: sorted() for Non-Destructive Sorting

# ❌ Bad - modifies original list
original = [3, 1, 4, 1, 5]
original.sort()
print(original)  # [1, 1, 3, 4, 5] - original destroyed!

# ✅ Good - preserves original
original = [3, 1, 4, 1, 5]
sorted_copy = sorted(original)
print(original)      # [3, 1, 4, 1, 5] - unchanged
print(sorted_copy)   # [1, 1, 3, 4, 5]

# Works with any iterable
sorted_set = sorted({3, 1, 4, 1, 5})  # [1, 3, 4, 5]
sorted_dict_keys = sorted({'z': 1, 'a': 2, 'm': 3})  # ['a', 'm', 'z']

Advanced Sorting Techniques

from operator import itemgetter, attrgetter, methodcaller

# 1. Sort objects by attribute
class Person:
    def __init__(self, name, age):
        self.name = name
        self.age = age
    def __repr__(self):
        return f"Person({self.name}, {self.age})"

people = [Person('Alice', 30), Person('Bob', 25), Person('Charlie', 35)]

# Sort by attribute
people.sort(key=attrgetter('age'))
# [Person(Bob, 25), Person(Alice, 30), Person(Charlie, 35)]

# Sort by multiple attributes
people.sort(key=attrgetter('age', 'name'))

# 2. Sort by method result
words = ['Python', 'java', 'C++', 'javascript']
words.sort(key=methodcaller('lower'))  # Case-insensitive sort
# ['C++', 'java', 'javascript', 'Python']

# 3. Reverse sorting (descending)
numbers = [3, 1, 4, 1, 5, 9]
numbers.sort(reverse=True)  # [9, 5, 4, 3, 1, 1]

# Or with key
people.sort(key=attrgetter('age'), reverse=True)  # Oldest first

# 4. Complex sorting with lambda (when operator functions won't work)
points = [(1, 5), (3, 2), (1, 3), (2, 4)]
# Sort by distance from origin
points.sort(key=lambda p: p[0]**2 + p[1]**2)

# But prefer operator when possible (faster):
points.sort(key=itemgetter(1))  # Sort by y-coordinate

Decorate-Sort-Undecorate (DSU) Pattern (Legacy)

# Old Python 2 pattern - now obsolete with key parameter
# But understanding it helps explain how key works internally

# ❌ Manual DSU (old way)
def sortby_manual(somelist, n):
    # Decorate
    decorated = [(x[n], x) for x in somelist]
    # Sort
    decorated.sort()
    # Undecorate
    return [x for (key, x) in decorated]

# ✅ Modern way (Python 2.4+)
def sortby_modern(somelist, n):
    return sorted(somelist, key=itemgetter(n))

Sorting Stability (Important!)

# Python's sort is STABLE (since 2.3)
# Equal elements maintain their relative order

students = [
    {'name': 'Alice', 'grade': 'A', 'age': 20},
    {'name': 'Bob', 'grade': 'B', 'age': 19},
    {'name': 'Charlie', 'grade': 'A', 'age': 21},
    {'name': 'David', 'grade': 'B', 'age': 20}
]

# Sort by grade (stable)
students.sort(key=itemgetter('grade'))
# Within same grade, original order preserved

# Multi-level sorting using stability
# Sort by secondary key first, then primary key
students.sort(key=itemgetter('age'))      # Sort by age first
students.sort(key=itemgetter('grade'))    # Then by grade (stable!)
# Result: Sorted by grade, and within same grade, sorted by age

Performance Comparison

import timeit
from operator import itemgetter

data = [{'id': i, 'value': i % 100} for i in range(10000)]

# Using lambda
time_lambda = timeit.timeit(
    lambda: sorted(data, key=lambda x: x['value']),
    number=1000
)

# Using itemgetter
time_itemgetter = timeit.timeit(
    lambda: sorted(data, key=itemgetter('value')),
    number=1000
)

print(f"Lambda: {time_lambda:.4f}s")
# Output: ~2.5s

print(f"itemgetter: {time_itemgetter:.4f}s")
# Output: ~1.8s (30-40% faster!)

When to Use What

# Use .sort() when:
# - You want to modify the list in place
# - You don't need the original order
# - Slightly more memory efficient
my_list.sort(key=itemgetter('field'))

# Use sorted() when:
# - You need to keep the original
# - Sorting any iterable (not just lists)
# - More functional programming style
new_list = sorted(my_list, key=itemgetter('field'))

# Use itemgetter when:
# - Sorting by dictionary keys or tuple/list indices
# - Need maximum performance
from operator import itemgetter
data.sort(key=itemgetter('age', 'name'))

# Use attrgetter when:
# - Sorting objects by attributes
from operator import attrgetter
objects.sort(key=attrgetter('attribute'))

# Use lambda when:
# - Complex transformation needed
# - itemgetter/attrgetter won't work
data.sort(key=lambda x: (x['category'], -x['priority']))

3. String Concatenation

❌ Avoid: Using += for String Building

# BAD - Creates a new string object on every iteration
def build_html_bad(items):
    html = ""
    for item in items:
        html += "<li>" + item + "</li>"  # Creates new string each time
    return html

# Why it's slow:
# Iteration 1: "" -> "<li>apple</li>" (new string created)
# Iteration 2: "<li>apple</li>" -> "<li>apple</li><li>banana</li>" (another new string)
# Each concatenation copies ALL previous characters again!

Why This is Slow:

Strings are immutable in Python
Each += creates a completely new string object
For n items, this copies characters O(n²) times
With 10,000 items, you might copy millions of characters

✅ Use: join() Method

# GOOD - Builds list first, then joins once
def build_html_good(items):
    parts = []
    for item in items:
        parts.append(f"<li>{item}</li>")
    return "".join(parts)

# Even better - list comprehension
def build_html_best(items):
    return "".join(f"<li>{item}</li>" for item in items)

Why This is Fast:

List operations are cheap
join() calculates total size once and allocates memory once
Only one string copy operation at the end
O(n) time complexity instead of O(n²)

Performance Comparison

import time

items = ['item' + str(i) for i in range(10000)]

# Bad approach
start = time.time()
result = ""
for item in items:
    result += item
print(f"Concatenation: {time.time() - start:.4f}s")
# Output: ~2-5 seconds

# Good approach
start = time.time()
result = "".join(items)
print(f"Join: {time.time() - start:.4f}s")
# Output: ~0.001 seconds (1000x faster!)

String Formatting

# ❌ Avoid concatenation
output = "<html>" + head + prologue + query + tail + "</html>"

# ✅ Use formatting (better)
output = "<html>%s%s%s%s</html>" % (head, prologue, query, tail)

# ✅ Use f-strings (Python 3.6+, best)
output = f"<html>{head}{prologue}{query}{tail}</html>"

3. Loops and Iteration

❌ Avoid: Manual Loop with Append

# BAD - Slow due to repeated attribute lookups and Python loop overhead
def process_words_bad(words):
    result = []
    for word in words:
        result.append(word.upper())
    return result

Why This is Slow:

Python interpreter overhead for each iteration
Repeated method lookups (.append, .upper)
Function call overhead for each operation

✅ Use: List Comprehensions

# GOOD - Optimized by interpreter
def process_words_good(words):
    return [word.upper() for word in words]

Why This is Fast:

List comprehensions are optimized at the bytecode level
Reduces interpreter overhead
More concise and readable

✅ Use: map() for Simple Operations

# ALSO GOOD - Pushes loop into C code
def process_words_map(words):
    return list(map(str.upper, words))

Why This is Fast:

map() is implemented in C
No Python interpreter overhead per iteration
Very efficient for simple transformations

Map in detail

The map() function in Python is used to apply a function to every item in an iterable (like a list or tuple) and return a map object (which can be converted to a list, set, etc.).

🧠 Syntax of `map()`

map(function, iterable)

function: A function that will be applied to each item in the iterable.
iterable: A sequence (like a list, tuple, etc.) whose items will be processed by the function.

✅ Example 1: Using Built-in Function `str.upper`

words = ["hello", "world"]
result = map(str.upper, words)
print(list(result))  # Output: ['HELLO', 'WORLD']

Here, str.upper is applied to each word in the list.

✅ Example 2: Using a Lambda Function

numbers = [1, 2, 3, 4]
squared = map(lambda x: x**2, numbers)
print(list(squared))  # Output: [1, 4, 9, 16]

The lambda function lambda x: x**2 squares each number.

✅ Example 3: Using a Custom Function

def add_prefix(word):
    return "pre_" + word

words = ["fix", "view", "dict"]
result = map(add_prefix, words)
print(list(result))  # Output: ['pre_fix', 'pre_view', 'pre_dict']

Generator Expressions (Memory Efficient)

# When you don't need the full list at once
def process_large_file(filename):
    with open(filename) as f:
        # ❌ Bad - loads entire file into memory
        lines = [line.upper() for line in f]

        # ✅ Good - processes one line at a time
        lines = (line.upper() for line in f)

        for line in lines:
            process(line)  # Only one line in memory at a time

Generator Expression in Detail

You're asking a great question — and it's a common point of confusion for many Python learners. Let's clarify the difference between yield and generators, and how they relate to each other.

🧠 What Is a Generator?

A generator is a special type of iterator in Python that produces values one at a time, only when requested. It’s useful for working with large datasets or streams of data because it doesn’t store everything in memory.

There are two ways to create a generator:

1. Using a Generator Function with `yield`

def my_generator():
    yield 1
    yield 2
    yield 3

When you call my_generator(), it returns a generator object.
Each time you iterate over it (e.g., with a for loop), it runs until it hits a yield, returns that value, and pauses.
When you ask for the next value, it resumes from where it left off.

2. Using a Generator Expression

gen = (x * x for x in range(3))

This is similar to a list comprehension, but with ()` instead of [].
It also returns a generator object and evaluates lazily (one item at a time).

🔄 What Is `yield`?

yield is a keyword used inside a function to turn it into a generator function.
It’s like return, but instead of ending the function, it pauses and allows the function to continue later.

✅ Example: Using `yield`

`python
def count_up_to(n):
count = 1
while count <= n:
yield count
count += 1

gen = count_up_to(3)
for num in gen:
print(num)
`

Output:

1 2 3

Each call to next(gen) gives the next number.
The function remembers its state between calls.

🔁 What a Generator Does

When you write:
python lines = (line.upper() for line in f)

This creates a generator object. It doesn’t actually read or process any lines yet. It just sets up the logic for how each line will be processed when requested.

🧠 Why the `for` Loop Is Needed

python for line in lines: process(line)

The for loop triggers** the generator to start reading the file line by line, converting each line to uppercase, and passing it to process().

Without the loop, the generator just sits there — it doesn’t do anything.

4. Avoiding Dots (Attribute Lookups)

❌ Avoid: Repeated Attribute Lookups

`python

BAD - Looks up .append and .upper on every iteration

def process_bad(words):
result = []
for word in words:
result.append(word.upper()) # Two lookups per iteration
return result
`

Why This is Slow:

Python does attribute lookup at runtime
Each dot (.) triggers a dictionary lookup
For 1 million items, that's 2 million dictionary lookups!

✅ Use: Cache Attribute Lookups

`python

GOOD - Lookup once, use many times

def process_good(words):
result = []
append = result.append # Cache the method
upper = str.upper # Cache the function

for word in words:
    append(upper(word))  # Direct reference, no lookup

return result

Why This is Fast:

Attribute lookup happens only once
Direct variable access is much faster
Reduces bytecode instructions per iteration

Real-World Example

`python

❌ Bad - repeated lookups

def parse_data_bad(data):
results = []
for item in data:
if item.value > 0: # Lookup 'value'
results.append({ # Lookup 'append'
'id': item.id, # Lookup 'id'
'name': item.name # Lookup 'name'
})
return results

✅ Good - cache lookups

def parse_data_good(data):
results = []
append = results.append

for item in data:
    value = item.value
    if value > 0:
        append({
            'id': item.id,
            'name': item.name
        })
return results

Caution: Only use this technique in performance-critical loops. It reduces readability, so use it judiciously.

5. Local vs Global Variables

❌ Avoid: Global Variables in Loops

`python

BAD - Accessing globals is slow

counter = 0

def process_global():
global counter
for i in range(1000000):
counter += 1 # Global lookup on every iteration
`

Why This is Slow:

Global variables are stored in a dictionary (globals())
Each access requires a dictionary lookup
Much slower than local variable access

✅ Use: Local Variables

`python

GOOD - Local variables use optimized storage

def process_local():
counter = 0 # Local variable
for i in range(1000000):
counter += 1 # Fast local access
return counter
`

Why This is Fast:

Local variables are stored in an array-like structure
Access is by index, not dictionary lookup
Much faster at the C level

Best Practice

`python

❌ Avoid

import math

def calculate_distances(points):
distances = []
for p1, p2 in points:
# math.sqrt is a global lookup each time
dist = math.sqrt((p2[0]-p1[0])2 + (p2[1]-p1[1])2)
distances.append(dist)
return distances

✅ Use

import math

def calculate_distances_fast(points):
distances = []
append = distances.append
sqrt = math.sqrt # Make it local!

for p1, p2 in points:
    dist = sqrt((p2[0]-p1[0])**2 + (p2[1]-p1[1])**2)
    append(dist)

return distances

6. Dictionary Initialization

❌ Avoid: if-else for Dictionary Keys

`python

BAD - Dictionary lookup happens twice on every iteration

def count_words_bad(words):
word_count = {}
for word in words:
if word not in word_count: # First lookup
word_count[word] = 0
word_count[word] += 1 # Second lookup
return word_count
`

Why This is Slow:

Double dictionary lookup for existing keys
if statement evaluated every single time
After first occurrence, the if always fails but still gets checked

✅ Use: try-except (EAFP)

`python

GOOD - Only one lookup for existing keys

def count_words_try(words):
word_count = {}
for word in words:
try:
word_count[word] += 1 # Try to increment
except KeyError:
word_count[word] = 1 # Only runs once per unique word
return word_count
`

Why This is Fast:

Python's EAFP (Easier to Ask for Forgiveness than Permission) philosophy
Exceptions are cheap when not raised
Only one dictionary lookup for existing keys
Exception only raised once per unique word

✅ Use: dict.get() with Default

`python

ALSO GOOD - Clear and concise

def count_words_get(words):
word_count = {}
for word in words:
word_count[word] = word_count.get(word, 0) + 1
return word_count
`

✅ Use: defaultdict (Best for Most Cases)

`python

BEST - Most Pythonic and readable

from collections import defaultdict

def count_words_defaultdict(words):
word_count = defaultdict(int) # int() returns 0
for word in words:
word_count[word] += 1 # No checking needed!
return word_count
`

Why This is Best:

No explicit initialization needed
Clear intent
Very efficient

7. Import Statement Overhead

❌ Avoid: Imports Inside Tight Loops

`python

BAD - Imports module on every function call

def process_data_bad():
for i in range(100000):
import string # Module lookup happens 100,000 times!
result = string.ascii_lowercase
`

Why This is Slow:

Python checks sys.modules on every import
Even though module isn't reloaded, the lookup is expensive
Adds unnecessary overhead to every iteration

✅ Use: Import at Module Level

`python

GOOD - Import once

import string

def process_data_good():
for i in range(100000):
result = string.ascii_lowercase # Direct access
`

Why This is Fast:

Module imported only once when file is loaded
No import overhead in the function
Variable lookup is much faster

✅ Use: Import Specific Names

`python

EVEN BETTER - No attribute lookup

from string import ascii_lowercase

def process_data_better():
for i in range(100000):
result = ascii_lowercase # No dot lookup!
`

Lazy Imports (When Needed) see import email part

`python

When module might not be needed

class EmailProcessor:
def init(self):
self._email_module = None

def parse_email(self, email_string):
    # ✅ Import only when first needed
    if self._email_module is None:
        import email
        self._email_module = email

    return self._email_module.message_from_string(email_string)

This is useful when:

1. Import is expensive (large module)

2. Module might not be used in this execution

3. You want faster startup time

8. Data Aggregation

❌ Avoid: Processing Items One at a Time

`python

BAD - Function call overhead for each item

def process_items_bad(items):
total = 0
for item in items:
total = add_to_total(total, item) # Function call per item
return total

def add_to_total(total, item):
return total + item
`

Why This is Slow:

Python function calls are expensive
Stack frame creation/destruction for each call
Parameter passing overhead
For 1 million items, 1 million function calls!

✅ Use: Process Data in Batches

`python

GOOD - Process entire collection at once

def process_items_good(items):
return sum(items) # Built-in, implemented in C

Or if you need custom processing

def process_items_batch(items):
total = 0
for item in items: # Loop inside the function
total += item
return total
`

Why This is Fast:

Single function call regardless of data size
Loop overhead amortized over all items
Built-in functions use optimized C code

Real-World Example

`python

❌ Bad - one API call per item

def save_users_bad(users):
for user in users:
database.save(user) # Network round-trip each time

✅ Good - batch API call

def save_users_good(users):
database.bulk_save(users) # Single network round-trip

❌ Bad - one validation per call

def validate_emails_bad(emails):
valid = []
for email in emails:
if is_valid_email(email): # Function call per email
valid.append(email)
return valid

✅ Good - validate in batch

def validate_emails_good(emails):
return [email for email in emails
if '@' in email and '.' in email] # Inline check
`

9. Advanced Built-in Functions and Techniques

all() and any() - Short-Circuit Evaluation

`python

❌ Bad - checks every element even after finding result

def has_negative_manual(numbers):
found = False
for num in numbers:
if num < 0:
found = True
break # Manual short-circuit
return found

✅ Good - built-in with automatic short-circuit

def has_negative_builtin(numbers):
return any(num < 0 for num in numbers)

Why it's better:

numbers = list(range(-1, 1000000))

any() stops at -1 (first element)

Manual loop in Python is slower even with break

Key Point: any() and all() are implemented in C and stop as soon as the result is determined.

`python

Real-world examples

def validate_data(records):
# Check if all records are valid (stops at first invalid)
return all(record.get('id') and record.get('name') for record in records)

def has_error(responses):
# Check if any response has error (stops at first error)
return any(resp.status_code >= 400 for resp in responses)
`

enumerate() - Better Than range(len())

`python

❌ Bad - manual indexing

items = ['apple', 'banana', 'cherry']
for i in range(len(items)):
print(f"{i}: {items[i]}") # Extra lookup

✅ Good - enumerate provides index and value

for i, item in enumerate(items):
print(f"{i}: {item}") # No lookup needed

Start from different index

for i, item in enumerate(items, start=1):
print(f"{i}: {item}") # 1: apple, 2: banana, 3: cherry
`

zip() - Parallel Iteration

`python

❌ Bad - manual indexing for parallel lists

names = ['Alice', 'Bob', 'Charlie']
ages = [30, 25, 35]
cities = ['NYC', 'LA', 'Chicago']

for i in range(len(names)):
print(f"{names[i]}, {ages[i]}, {cities[i]}")

✅ Good - zip combines iterables

for name, age, city in zip(names, ages, cities):
print(f"{name}, {age}, {city}")

Create dictionary from two lists

keys = ['a', 'b', 'c']
values = [1, 2, 3]
dict_from_zip = dict(zip(keys, values)) # {'a': 1, 'b': 2, 'c': 3}

Unzip (transpose)

pairs = [(1, 'a'), (2, 'b'), (3, 'c')]
numbers, letters = zip(*pairs)

numbers = (1, 2, 3), letters = ('a', 'b', 'c')

itertools - The Power Tools

`python
from itertools import chain, islice, groupby, accumulate, product, combinations

1. chain - flatten multiple iterables (no intermediate list!)

❌ Bad

list1 = [1, 2, 3]
list2 = [4, 5, 6]
combined = list1 + list2 # Creates new list

✅ Good

combined = chain(list1, list2) # Lazy iterator
for item in combined:
print(item) # No intermediate list created

2. islice - slicing iterators without loading everything

❌ Bad - loads entire file

with open('huge_file.txt') as f:
first_10 = list(f)[:10] # Loads entire file!

✅ Good - only reads what's needed

from itertools import islice
with open('huge_file.txt') as f:
first_10 = list(islice(f, 10)) # Reads only 10 lines

3. groupby - group consecutive items

from operator import itemgetter
data = [
{'category': 'A', 'value': 1},
{'category': 'A', 'value': 2},
{'category': 'B', 'value': 3},
{'category': 'B', 'value': 4}
]
data.sort(key=itemgetter('category')) # MUST be sorted first!

for category, items in groupby(data, key=itemgetter('category')):
print(f"{category}: {list(items)}")

4. accumulate - running totals

from itertools import accumulate
numbers = [1, 2, 3, 4, 5]
running_sum = list(accumulate(numbers)) # [1, 3, 6, 10, 15]

Custom operation

running_product = list(accumulate(numbers, lambda x, y: x * y))

[1, 2, 6, 24, 120]

5. product - cartesian product (nested loops)

❌ Bad - manual nested loops

result = []
for color in ['red', 'blue']:
for size in ['S', 'M', 'L']:
result.append((color, size))

✅ Good - itertools.product

from itertools import product
result = list(product(['red', 'blue'], ['S', 'M', 'L']))

[('red', 'S'), ('red', 'M'), ('red', 'L'), ('blue', 'S'), ...]

6. combinations and permutations

from itertools import combinations, permutations
items = ['A', 'B', 'C']

All 2-item combinations (order doesn't matter)

list(combinations(items, 2))

[('A', 'B'), ('A', 'C'), ('B', 'C')]

All 2-item permutations (order matters)

list(permutations(items, 2))

[('A', 'B'), ('A', 'C'), ('B', 'A'), ('B', 'C'), ('C', 'A'), ('C', 'B')]

set Operations - O(1) Membership Testing

`python

❌ Bad - O(n) lookup in list

allowed_users = ['alice', 'bob', 'charlie'] # List
for user in all_users:
if user in allowed_users: # O(n) check!
process(user)

✅ Good - O(1) lookup in set

allowed_users = {'alice', 'bob', 'charlie'} # Set
for user in all_users:
if user in allowed_users: # O(1) check!
process(user)

Set operations

set1 = {1, 2, 3, 4, 5}
set2 = {4, 5, 6, 7, 8}

Intersection (common elements)

common = set1 & set2 # {4, 5}

Union (all elements)

all_elements = set1 | set2 # {1, 2, 3, 4, 5, 6, 7, 8}

Difference (in set1 but not set2)

only_in_set1 = set1 - set2 # {1, 2, 3}

Symmetric difference (in either but not both)

symmetric = set1 ^ set2 # {1, 2, 3, 6, 7, 8}

Remove duplicates while preserving order (Python 3.7+)

items = [1, 2, 2, 3, 1, 4, 3, 5]
unique = list(dict.fromkeys(items)) # [1, 2, 3, 4, 5]
`

functools - Function Tools

`python
from functools import lru_cache, partial, reduce

1. lru_cache - Memoization (caching results)

❌ Bad - recalculates same values

def fibonacci_slow(n):
if n < 2:
return n
return fibonacci_slow(n-1) + fibonacci_slow(n-2)

fibonacci_slow(35) takes ~5 seconds

✅ Good - caches results

@lru_cache(maxsize=128)
def fibonacci_fast(n):
if n < 2:
return n
return fibonacci_fast(n-1) + fibonacci_fast(n-2)

fibonacci_fast(35) takes ~0.0001 seconds!

Real-world example: expensive API call

@lru_cache(maxsize=100)
def get_user_data(user_id):
# Expensive database query or API call
return database.query(user_id)

2. partial - Pre-fill function arguments

def power(base, exponent):
return base ** exponent

square = partial(power, exponent=2)
cube = partial(power, exponent=3)

print(square(5)) # 25
print(cube(5)) # 125

Useful with map/filter

from functools import partial
from operator import mul

double = partial(mul, 2)
numbers = [1, 2, 3, 4, 5]
doubled = list(map(double, numbers)) # [2, 4, 6, 8, 10]

3. reduce - Cumulative operations

from functools import reduce
from operator import mul

numbers = [1, 2, 3, 4, 5]
product = reduce(mul, numbers) # 1*2*3*4*5 = 120

sum() instead of reduce(add, numbers)

math.prod() (Python 3.8+) instead of reduce(mul, numbers)

collections Module Power Tools

`python
from collections import deque, namedtuple, ChainMap

1. deque - Fast appends/pops from both ends

❌ Bad - list is O(n) for left operations

my_list = [1, 2, 3]
my_list.insert(0, 0) # O(n) - shifts all elements

✅ Good - deque is O(1) for both ends

from collections import deque
my_deque = deque([1, 2, 3])
my_deque.appendleft(0) # O(1)
my_deque.append(4) # O(1)
my_deque.popleft() # O(1)
my_deque.pop() # O(1)

Ring buffer / sliding window

recent_items = deque(maxlen=5) # Only keeps last 5 items
for item in range(10):
recent_items.append(item)
print(recent_items) # deque([5, 6, 7, 8, 9])

2. namedtuple - Lightweight object

❌ Bad - dictionary overhead

user = {'name': 'Alice', 'age': 30, 'email': 'alice@example.com'}
print(user['name']) # Dictionary lookup

✅ Good - namedtuple is faster and cleaner

from collections import namedtuple
User = namedtuple('User', ['name', 'age', 'email'])
user = User('Alice', 30, 'alice@example.com')
print(user.name) # Attribute access (faster)
print(user[0]) # Also supports indexing

Immutable and memory efficient

Uses tuple's memory layout but with named access

3. ChainMap - Combine multiple dictionaries

❌ Bad - creates new dictionary

config = {**defaults, **user_config, **override} # Memory overhead

✅ Good - ChainMap provides view without copying

from collections import ChainMap
combined = ChainMap(override, user_config, defaults)

Lookups check each dict in order, no copying

Context Managers - Resource Management

`python

❌ Bad - manual resource management

file = open('data.txt', 'r')
try:
data = file.read()
process(data)
finally:
file.close() # Easy to forget!

✅ Good - automatic cleanup

with open('data.txt', 'r') as file:
data = file.read()
process(data)

File automatically closed even if exception occurs

Multiple context managers

with open('input.txt') as infile, open('output.txt', 'w') as outfile:
for line in infile:
outfile.write(process(line))

Custom context manager with contextlib

from contextlib import contextmanager
import time

@contextmanager
def timer(name):
start = time.time()
yield # Control returns to caller
end = time.time()
print(f"{name} took {end-start:.4f}s")

Usage

with timer("Data processing"):
process_large_dataset()
`

Assignment Expressions (Walrus Operator) - Python 3.8+

`python

❌ Bad - compute twice or extra variable

data = get_data()
if len(data) > 0:
process(data)

Or

if len(get_data()) > 0: # Calls get_data() twice!
process(get_data())

✅ Good - compute once, use in condition

if (n := len(data)) > 0:
print(f"Processing {n} items")
process(data)

Useful in list comprehensions

❌ Bad - expensive_function called twice per item

results = [expensive_function(x) for x in items
if expensive_function(x) > threshold]

✅ Good - called once per item

results = [y for x in items
if (y := expensive_function(x)) > threshold]

While loop pattern

❌ Bad

line = file.readline()
while line:
process(line)
line = file.readline()

✅ Good

while (line := file.readline()):
process(line)
`

Great! Let's break down this line of Python code and explain it in detail:

🔍 Code in Question

python if (n := len(data)) > 0: print(f"Processing {n} items") process(data)

✅ What’s Happening Here?

This line uses the walrus operator :=, which was introduced in Python 3.8. It allows you to assign a value to a variable as part of an expression — especially useful in conditions.

🧠 Step-by-Step Explanation

len(data):
- Calculates the number of items in the data list (or any iterable).
n := len(data):
- Assigns the result of len(data) to the variable n.
- This is done inside the if condition, so you don’t need a separate line like: python n = len(data) if n > 0:
if (n := len(data)) > 0::
- Checks if the length of data is greater than 0.
- If true, it enters the block and uses n (already computed) without recalculating len(data).
print(f"Processing {n} items"):
- Prints how many items are being processed.
process(data):
- Calls a function named process() and passes the data to it.

📌 Why Is This Good Practice?

Efficiency: You compute len(data) once, but use it twice — in the condition and in the print statement.
Cleaner Code: Avoids repetition and keeps logic compact.
Readability: Once you're familiar with the walrus operator, it makes code more expressive.

🧪 Example Without Walrus Operator

python n = len(data) if n > 0: print(f"Processing {n} items") process(data)

This is perfectly fine too — but the walrus operator lets you do it in one line.

10. Profiling: Finding the Real Bottlenecks

Why Profile?

`python

You might think THIS is slow:

def process_data(data):
# Complex calculation
results = [x*2 + x3 + x*4 for x in data] # Suspicious!

# Simple operation
return '\n'.join(map(str, results))  # Looks innocent

But profiling reveals the truth:

95% of time is spent in join()!

Only 5% in the calculation

Basic Profiling with cProfile

`python
import cProfile
import pstats

def my_function():
# Your code here
data = list(range(10000))
result = [x**2 for x in data]
return sum(result)

Profile the function

cProfile.run('my_function()', 'output.prof')

Analyze results

stats = pstats.Stats('output.prof')
stats.sort_stats('cumulative')
stats.print_stats(10) # Top 10 slowest
`

Line-by-Line Profiling

`python

Install: pip install line_profiler

Add @profile decorator

@profile
def slow_function():
result = []
for i in range(10000):
result.append(i ** 2) # How slow is this?

total = sum(result)  # How about this?

return total

Run: kernprof -l -v script.py

Shows time spent on EACH LINE

Memory Profiling

`python

Install: pip install memory_profiler

from memory_profiler import profile

@profile
def memory_hog():
# ❌ Bad - creates huge list
big_list = [i for i in range(10000000)]

# ✅ Good - uses generator
big_gen = (i for i in range(10000000))

return sum(big_gen)

Run: python -m memory_profiler script.py

Quick Timing with timeit

`python
import timeit

Compare different approaches

setup = "data = list(range(1000))"

Approach 1

time1 = timeit.timeit(
"result = [x**2 for x in data]",
setup=setup,
number=10000
)

Approach 2

time2 = timeit.timeit(
"result = list(map(lambda x: x**2, data))",
setup=setup,
number=10000
)

print(f"List comp: {time1:.4f}s")
print(f"Map: {time2:.4f}s")
print(f"Winner: {'List comp' if time1 < time2 else 'Map'}")
`

13. Memory Optimization Techniques

Generators vs Lists

`python

❌ Bad - loads everything into memory

def read_large_file_bad(filename):
with open(filename) as f:
return [line.strip() for line in f] # All lines in memory!

For 1GB file, needs 1GB+ RAM

lines = read_large_file_bad('huge.txt')
for line in lines:
process(line)

✅ Good - processes one line at a time

def read_large_file_good(filename):
with open(filename) as f:
for line in f: # Generator, not list!
yield line.strip()

Only one line in memory at a time

for line in read_large_file_good('huge.txt'):
process(line)

Generator expression vs list comprehension

❌ List comprehension - immediate memory allocation

squares_list = [x**2 for x in range(1000000)] # ~8MB memory

✅ Generator expression - lazy evaluation

squares_gen = (x**2 for x in range(1000000)) # ~128 bytes

Values computed on-demand

When to Use Generators:

Processing large datasets
Pipeline operations (one result feeds into another)
You don't need the entire result set at once
Memory is a concern

When to Use Lists:

Need multiple passes over data
Need random access (indexing)
Need to know the length
Dataset is small

slots - Reduce Object Memory

`python

❌ Bad - each instance has dict (overhead)

class Point:
def init(self, x, y):
self.x = x
self.y = y

Each instance: ~280 bytes (with dict)

p = Point(1, 2)
print(p.dict) # {'x': 1, 'y': 2}

✅ Good - use slots for memory efficiency

class PointSlots:
slots = ['x', 'y'] # Fixed attributes

def __init__(self, x, y):
    self.x = x
    self.y = y

Each instance: ~48 bytes (no dict)

~80% memory reduction!

Real impact with many objects

1 million Points: ~280MB vs ~48MB

points = [PointSlots(i, i) for i in range(1000000)]
`

When to Use slots:

Creating millions of instances
Objects have fixed set of attributes
Memory is critical (e.g., data science, games)

Tradeoffs:

Can't add attributes dynamically
Slightly less flexible
No dict attribute
Can't use with multiple inheritance (complex rules)

Array Module - Compact Numeric Arrays

`python

❌ Bad - list of numbers (lots of overhead)

numbers_list = [1, 2, 3, 4, 5] * 100000

Each integer object: ~28 bytes

500,000 integers: ~14MB

✅ Good - array module (C-style array)

from array import array
numbers_array = array('i', [1, 2, 3, 4, 5] * 100000)

Each integer: 4 bytes

500,000 integers: ~2MB (7x less memory!)

Type codes:

'b': signed char (1 byte)

'i': signed int (4 bytes)

'f': float (4 bytes)

'd': double (8 bytes)

For numerical computing, use numpy (even better)

import numpy as np
numbers_numpy = np.array([1, 2, 3, 4, 5] * 100000, dtype=np.int32)

Optimized for mathematical operations

String Interning

`python

Python automatically interns some strings

a = "hello"
b = "hello"
print(a is b) # True - same object in memory

Force interning for frequently used strings

from sys import intern

❌ Bad - many identical strings

tags = ["python", "python", "python"] * 10000

Each "python" might be a separate object

✅ Good - intern frequently used strings

tags = [intern("python")] * 30000

All reference same object

Useful for:

- Large datasets with repeated string values

- Dictionary keys that repeat

- Tag systems, category labels

Weak References - Avoid Circular References

`python
import weakref

❌ Bad - circular reference prevents garbage collection

class Node:
def init(self, value):
self.value = value
self.parent = None
self.children = []

def add_child(self, child):
    child.parent = self  # Strong reference cycle!
    self.children.append(child)

Creates memory leak - nodes never freed

✅ Good - use weak references

class NodeWeak:
def init(self, value):
self.value = value
self._parent = None
self.children = []

@property
def parent(self):
    return self._parent() if self._parent else None

@parent.setter
def parent(self, node):
    self._parent = weakref.ref(node) if node else None

def add_child(self, child):
    child.parent = self  # Weak reference, can be freed
    self.children.append(child)

14. Algorithm Complexity Matters

Choose the Right Data Structure

`python

Searching: O(n) vs O(1)

❌ Bad - O(n) lookup in list

valid_ids = [1, 5, 10, 15, 20, 25, 30] # 7 items
if user_id in valid_ids: # Checks each item: O(n)
grant_access()

✅ Good - O(1) lookup in set

valid_ids = {1, 5, 10, 15, 20, 25, 30}
if user_id in valid_ids: # Hash lookup: O(1)
grant_access()

For 1 million IDs, 1 million lookups:

List: ~trillion operations

Set: ~1 million operations (1000x faster!)

Removing duplicates

❌ Bad - O(n²)

def remove_duplicates_bad(items):
result = []
for item in items:
if item not in result: # O(n) check
result.append(item)
return result

10,000 items: ~100 million operations

✅ Good - O(n)

def remove_duplicates_good(items):
return list(dict.fromkeys(items)) # Preserves order (3.7+)

10,000 items: ~10,000 operations

Or if order doesn't matter:

def remove_duplicates_set(items):
return list(set(items))
`

Avoid Nested Loops When Possible

`python

❌ Bad - O(n²) complexity

def find_common_bad(list1, list2):
common = []
for item1 in list1: # O(n)
for item2 in list2: # O(m)
if item1 == item2:
common.append(item1)
return common

1000 items each: 1 million comparisons

✅ Good - O(n + m) complexity

def find_common_good(list1, list2):
set2 = set(list2) # O(m)
return [item for item in list1 if item in set2] # O(n)

1000 items each: ~2000 operations (500x faster!)

Even better - use set intersection

def find_common_best(list1, list2):
return list(set(list1) & set(list2)) # O(min(n, m))

Practical example: finding duplicate emails

❌ Bad - O(n²)

def find_duplicate_emails_bad(users):
duplicates = []
for i, user1 in enumerate(users):
for user2 in users[i+1:]:
if user1['email'] == user2['email']:
duplicates.append(user1['email'])
return duplicates

✅ Good - O(n)

def find_duplicate_emails_good(users):
seen = set()
duplicates = set()
for user in users:
email = user['email']
if email in seen:
duplicates.add(email)
else:
seen.add(email)
return list(duplicates)
`

Use bisect for Sorted Data

`python
import bisect

❌ Bad - linear search in sorted list O(n)

sorted_numbers = list(range(0, 1000000, 2)) # Even numbers

def find_position_bad(numbers, value):
for i, num in enumerate(numbers):
if num >= value:
return i
return len(numbers)

✅ Good - binary search O(log n)

def find_position_good(numbers, value):
return bisect.bisect_left(numbers, value)

Insert while maintaining sort order

❌ Bad - O(n log n) due to sort

def insert_sorted_bad(numbers, value):
numbers.append(value)
numbers.sort()

✅ Good - O(n) just for insertion

def insert_sorted_good(numbers, value):
bisect.insort(numbers, value)

Real-world example: maintaining sorted timestamps

from bisect import insort
timestamps = []

def add_event(event_time):
insort(timestamps, event_time) # Keeps sorted

def get_events_in_range(start, end):
# O(log n) to find positions
left = bisect.bisect_left(timestamps, start)
right = bisect.bisect_right(timestamps, end)
return timestamps[left:right]
`

15. String Operations Optimization

String Methods vs Regex

`python
import re
import time

text = "Hello, World! This is a test." * 1000

❌ Slower - regex for simple operations

start = time.time()
for _ in range(10000):
result = re.sub(r'test', 'exam', text)
print(f"Regex: {time.time() - start:.4f}s")

Output: ~0.8s

✅ Faster - string method

start = time.time()
for _ in range(10000):
result = text.replace('test', 'exam')
print(f"String method: {time.time() - start:.4f}s")

Output: ~0.1s (8x faster!)

Use regex only when you need pattern matching

❌ Overkill

if re.match(r'hello', text.lower()):
pass

✅ Better

if text.lower().startswith('hello'):
pass

When regex is appropriate:

- Pattern matching: r'\d{3}-\d{3}-\d{4}' for phone numbers

- Complex replacements: r'(\w+)\s+(\w+)' -> r'\2 \1'

- Multiple patterns: r'(cat|dog|bird)'

String Building with f-strings (Python 3.6+)

`python

Performance comparison

name = "Alice"
age = 30
city = "NYC"

❌ Slowest - concatenation

result = "Name: " + name + ", Age: " + str(age) + ", City: " + city

✅ Fast - % formatting

result = "Name: %s, Age: %d, City: %s" % (name, age, city)

✅ Fast - .format()

result = "Name: {}, Age: {}, City: {}".format(name, age, city)

✅ Fastest - f-strings (and most readable!)

result = f"Name: {name}, Age: {age}, City: {city}"

f-strings can also include expressions

prices = [10.50, 23.75, 5.25]
result = f"Total: ${sum(prices):.2f}" # Total: $39.50

Multi-line f-strings

message = (
f"User: {name}\n"
f"Age: {age}\n"
f"City: {city}"
)
`

str.translate() - Fast Character Replacement

`python

❌ Bad - multiple replace calls

def remove_punctuation_bad(text):
for char in '.,;:!?':
text = text.replace(char, '')
return text

✅ Good - single translate call

def remove_punctuation_good(text):
translator = str.maketrans('', '', '.,;:!?')
return text.translate(translator)

Even more complex translations

def leetspeak(text):
translation_table = str.maketrans({
'a': '4',
'e': '3',
'i': '1',
'o': '0',
's': '5',
't': '7'
})
return text.lower().translate(translation_table)

print(leetspeak("Hello World")) # h3ll0 w0rld

Performance comparison

import time
text = "Hello, World! How are you today?" * 10000

start = time.time()
for _ in range(1000):
result = remove_punctuation_bad(text)
print(f"Multiple replace: {time.time() - start:.4f}s")

Output: ~1.2s

start = time.time()
for _ in range(1000):
result = remove_punctuation_good(text)
print(f"Translate: {time.time() - start:.4f}s")

Output: ~0.15s (8x faster!)

16. Comprehensions vs map/filter

When to Use Each

`python
numbers = range(1000)

List comprehension - clear and Pythonic

squares = [x**2 for x in numbers]

map with lambda - slower due to lambda

squares = list(map(lambda x: x**2, numbers))

map with built-in function - FASTEST

from operator import mul
from functools import partial
double = partial(mul, 2)
doubled = list(map(double, numbers))

✅ Use list comprehension when:

- Transformation is complex

- Need filtering with transformation

- Readability is priority

result = [x**2 for x in numbers if x % 2 == 0]

✅ Use map when:

- Applying built-in function or C-function

- Don't need intermediate list (return iterator)

result = map(str.upper, words) # Iterator, no list created

✅ Use filter when:

- Simple predicate function

- Don't need transformation

result = filter(str.isdigit, characters)
`

Nested Comprehensions

`python

❌ Bad - hard to read

matrix = []
for i in range(3):
row = []
for j in range(3):
row.append(i * j)
matrix.append(row)

✅ Good - nested comprehension

matrix = [[i * j for j in range(3)] for i in range(3)]

[[0, 0, 0], [0, 1, 2], [0, 2, 4]]

Flatten nested list

❌ Bad - manual loops

nested = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
flat = []
for sublist in nested:
for item in sublist:
flat.append(item)

✅ Good - list comprehension

flat = [item for sublist in nested for item in sublist]

[1, 2, 3, 4, 5, 6, 7, 8, 9]

✅ Best for deeply nested - use itertools

from itertools import chain
flat = list(chain.from_iterable(nested))

Dictionary comprehension

❌ Bad

word_lengths = {}
for word in words:
word_lengths[word] = len(word)

✅ Good

word_lengths = {word: len(word) for word in words}

Set comprehension

❌ Bad

unique_lengths = set()
for word in words:
unique_lengths.add(len(word))

✅ Good

unique_lengths = {len(word) for word in words}
`

17. Number Operations Optimization

Integer Operations

`python

In Python, some operations are faster than others

❌ Slower

x = 47
result = x * 2 # Multiplication

✅ Faster (though less readable)

result = x << 1 # Bit shift (left shift by 1 = multiply by 2)

BUT: In practice, the difference is tiny

Only optimize this in extremely tight loops

Readability usually trumps micro-optimization

Division: // (floor division) vs / (true division)

import time

True division (returns float)

start = time.time()
for i in range(10000000):
result = i / 2
print(f"True division: {time.time() - start:.4f}s")

Output: ~0.5s

Floor division (returns int)

start = time.time()
for i in range(10000000):
result = i // 2
print(f"Floor division: {time.time() - start:.4f}s")

Output: ~0.4s (20% faster if you need integer result)

Modulo operations

Use & for power-of-2 modulo (much faster)

❌ Slower

if x % 2 == 0: # Check if even
pass

✅ Faster (but less clear)

if x & 1 == 0: # Check if even using bitwise AND
pass
`

Float Operations

`python

Avoid float when integers work

❌ Slower - unnecessary float operations

total = 0.0
for i in range(1000000):
total += float(i)

✅ Faster - integer operations

total = 0
for i in range(1000000):
total += i

Use math module for complex operations

import math

❌ Slower - repeated exponentiation

result = x ** 0.5

✅ Faster - dedicated function

result = math.sqrt(x)

❌ Slower

result = x ** 2

✅ Faster (and clearer)

result = x * x
`

Decimal for Financial Calculations

`python

❌ Bad - float precision issues

price = 0.1
quantity = 3
total = price * quantity
print(total) # 0.30000000000000004 (WRONG for money!)

✅ Good - Decimal for exact precision

from decimal import Decimal

price = Decimal('0.1')
quantity = 3
total = price * quantity
print(total) # 0.3 (CORRECT)

Real-world example

from decimal import Decimal, ROUND_HALF_UP

def calculate_total(prices):
total = sum(Decimal(str(price)) for price in prices)
# Round to 2 decimal places for currency
return total.quantize(Decimal('0.01'), rounding=ROUND_HALF_UP)

prices = [10.10, 20.20, 30.33]
print(calculate_total(prices)) # 60.63
`

18. Exception Handling Performance

EAFP vs LBYL

`python

LBYL (Look Before You Leap)

❌ Slower when condition is usually true

if key in dictionary:
value = dictionary[key]
else:
value = default

EAFP (Easier to Ask for Forgiveness than Permission)

✅ Faster when exception is rare

try:
value = dictionary[key]
except KeyError:
value = default

Why EAFP is faster:

- No extra dictionary lookup when key exists

- Exceptions are cheap when not raised

- Only pays penalty on actual error

Performance comparison

import time

data = {i: i*2 for i in range(10000)}

LBYL approach

start = time.time()
for i in range(10000):
if i in data: # Extra lookup
value = data[i] # Second lookup
print(f"LBYL: {time.time() - start:.4f}s")

Output: ~0.003s

EAFP approach

start = time.time()
for i in range(10000):
try:
value = data[i] # Single lookup
except KeyError:
pass
print(f"EAFP: {time.time() - start:.4f}s")

Output: ~0.002s (30% faster when key exists)

Avoid Exceptions in Hot Paths

`python

When exceptions are common, they're expensive

❌ Bad - exception raised 50% of the time

def process_items_bad(items):
results = []
for item in items:
try:
results.append(expensive_operation(item))
except ValueError: # Raised for 50% of items!
results.append(None)
return results

✅ Good - check first if exceptions are common

def process_items_good(items):
results = []
for item in items:
if is_valid(item): # Cheap check
results.append(expensive_operation(item))
else:
results.append(None)
return results
`

19. Parallel Processing

Threading vs Multiprocessing vs AsyncIO

`python
import time
import threading
import multiprocessing
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor

def cpu_bound_task(n):
"""Simulates CPU-intensive work"""
return sum(i*i for i in range(n))

def io_bound_task(n):
"""Simulates I/O wait (network, disk)"""
time.sleep(0.1) # Simulate I/O wait
return n * 2

❌ Bad - sequential for I/O-bound tasks

def process_io_sequential(tasks):
results = []
for task in tasks:
results.append(io_bound_task(task))
return results

Time: 10 tasks * 0.1s = 1 second

✅ Good - threading for I/O-bound

def process_io_threaded(tasks):
with ThreadPoolExecutor(max_workers=10) as executor:
results = list(executor.map(io_bound_task, tasks))
return results

Time: ~0.1 seconds (10x faster!)

❌ Bad - threading for CPU-bound (GIL limitation)

def process_cpu_threaded(tasks):
with ThreadPoolExecutor(max_workers=4) as executor:
results = list(executor.map(cpu_bound_task, tasks))
return results

No speedup due to Global Interpreter Lock (GIL)

✅ Good - multiprocessing for CPU-bound

def process_cpu_multiprocessing(tasks):
with ProcessPoolExecutor(max_workers=4) as executor:
results = list(executor.map(cpu_bound_task, tasks))
return results

Time: ~1/4 on quad-core processor

Use cases:

Threading: I/O-bound (network requests, file I/O, database queries)

Multiprocessing: CPU-bound (calculations, data processing, image processing)

AsyncIO: Many concurrent I/O operations (thousands of connections)

AsyncIO for High Concurrency

`python
import asyncio
import aiohttp # pip install aiohttp

❌ Bad - sequential HTTP requests

import requests

def fetch_urls_sequential(urls):
results = []
for url in urls:
response = requests.get(url)
results.append(response.text)
return results

10 URLs, 0.5s each = 5 seconds

✅ Good - async HTTP requests

async def fetch_url(session, url):
async with session.get(url) as response:
return await response.text()

async def fetch_urls_async(urls):
async with aiohttp.ClientSession() as session:
tasks = [fetch_url(session, url) for url in urls]
results = await asyncio.gather(*tasks)
return results

Run async function

results = asyncio.run(fetch_urls_async(urls))

10 URLs concurrently = ~0.5 seconds (10x faster!)

When to use async:

- Many concurrent I/O operations

- Web scraping (many URLs)

- API calls to multiple services

- Database queries (with async driver)

- WebSocket connections

20. Pro Tips and Best Practices

Use Virtual Environments

`bash

Always use virtual environments

python -m venv venv
source venv/bin/activate # Linux/Mac

venv\Scripts\activate # Windows

Why:

- Isolates dependencies

- Prevents version conflicts

- Reproducible environments

Utilize Python's Standard Library

`python

Standard library is optimized and battle-tested

Don't reinvent the wheel!

❌ Bad - manual date parsing

def parse_date_bad(date_string):
parts = date_string.split('-')
year = int(parts[0])
month = int(parts[1])
day = int(parts[2])
# ... validation logic ...
return (year, month, day)

✅ Good - use datetime

from datetime import datetime

def parse_date_good(date_string):
return datetime.strptime(date_string, '%Y-%m-%d')

Other powerful standard library modules:

- collections: defaultdict, Counter, deque, namedtuple

- itertools: chain, groupby, combinations, product

- functools: lru_cache, partial, reduce

- operator: itemgetter, attrgetter, methodcaller

- pathlib: modern path handling

- dataclasses: reduce boilerplate (Python 3.7+)

Type Hints and Static Analysis

`python

Use type hints for better tooling and catching errors

from typing import List, Dict, Optional, Union

def process_users(users: List[Dict[str, Union[str, int]]]) -> Optional[Dict[str, int]]:
"""Process users and return statistics.

Args:

    users: List of user dictionaries

Returns:

    Dictionary of statistics or None if empty

"""

if not users:

    return None

return {

    'total': len(users),

    'average_age': sum(u['age'] for u in users) // len(users)

}

Benefits:

- IDEs provide better autocomplete

- Catch type errors before runtime

- Self-documenting code

- Use mypy for static type checking: mypy script.py

Dataclasses (Python 3.7+)

`python

❌ Bad - lots of boilerplate

class PointOld:
def init(self, x, y):
self.x = x
self.y = y

def repr(self):

    return f"Point(x={self.x}, y={self.y})"

def eq(self, other):

    return self.x == other.x and self.y == other.y

✅ Good - dataclass handles boilerplate

from dataclasses import dataclass, field
from typing import List

@dataclass
class Point:
x: float
y: float

def distance(self) -> float:

    return (self.x*2 + self.y2) * 0.5

Auto-generates: init, repr, eq, and more!

@dataclass
class User:
name: str
age: int
emails: List[str] = field(default_factory=list) # Mutable default
active: bool = True # Simple default

Usage

user = User("Alice", 30)
print(user) # User(name='Alice', age=30, emails=[], active=True)
`

Use Logging, Not Print

`python

❌ Bad - debugging with print

def process_data(data):
print(f"Processing {len(data)} items") # Can't disable!
result = expensive_operation(data)
print(f"Result: {result}") # Clutters output
return result

✅ Good - use logging

import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(name)

def process_data(data):
logger.debug(f"Processing {len(data)} items")
result = expensive_operation(data)
logger.info(f"Processing complete")
return result

Benefits:

- Can adjust verbosity (DEBUG, INFO, WARNING, ERROR)

- Can log to files, not just console

- Can disable in production

- Includes timestamps and context

Summary: The Golden Rules of Python Performance

1. Measure First, Optimize Later

`python

Always profile before optimizing

import cProfile
cProfile.run('your_function()')
`

2. Use the Right Data Structure

List: Ordered, allows duplicates, O(n) membership
Tuple: Immutable list, slightly faster, can be dict key
Set: Unordered, no duplicates, O(1) membership
Dict: Key-value pairs, O(1) lookup
deque: Fast appends/pops from both ends
defaultdict: Dict with default values
Counter: Count hashable objects

3. Leverage Built-ins

`python

Built-ins are optimized C code

sum(numbers) # vs manual loop
any(conditions) # vs manual check
all(conditions) # vs manual check
max(numbers) # vs manual comparison
sorted(items) # vs manual sort
`

4. Avoid Premature Optimization

`python

❌ Don't optimize this (runs once)

config = eval(open('config.txt').read()) # Simple is fine

✅ Do optimize this (runs millions of times)

def hot_path(data):
# This needs optimization
pass
`

5. Write Pythonic Code

`python

Pythonic code is often faster AND more readable

Use comprehensions, iterators, context managers, etc.

6. When All Else Fails

Use Cython to compile Python to C
Use NumPy for numerical operations
Use PyPy JIT compiler
Rewrite bottlenecks in C/Rust
Use numba JIT compilation for numerical code

Final Wisdom

"Premature optimization is the root of all evil." - Donald Knuth

The best code is:

Correct first
Clear and maintainable
Fast enough for your needs

Only optimize what profiling shows is actually slow!# Python Performance Optimization: Detailed Guide

Summary: The Golden Rules

Profile First: Don't optimize without measuring
Use Built-ins: They're implemented in C and heavily optimized
Avoid String Concatenation: Use join() instead
Cache Lookups: Store frequently accessed attributes in local variables
Use Local Variables: Much faster than globals
List Comprehensions: Usually faster and more readable than loops
EAFP over LYBYL: try-except is often faster than if checks
Import Wisely: Keep imports out of loops
Batch Operations: Process data in aggregate, not item-by-item
Read the Docs: Standard library has optimized solutions for common problems

When to Optimize

`python

❌ Don't optimize this (runs once)

def load_config():
config = {}
for line in open('config.txt'):
# Not worth optimizing
key, value = line.split('=')
config[key] = value
return config

✅ DO optimize this (runs millions of times)

def process_requests(requests):
results = []
append = results.append # Worth optimizing
for request in requests:
append(expensive_operation(request))
return results
`

Remember: Readable code that runs fast enough is better than optimized code that's hard to maintain!

Langchain and LangGraph

Ashutosh Sarangi — Mon, 29 Sep 2025 07:02:50 +0000

LangChain: The Comprehensive LLM Application Framework

LangChain is a framework that simplifies the creation of applications using LLMs. It provides a modular and composable way to build complex LLM workflows by offering various components that can be chained together.

Key Concepts and Components of LangChain:

Models:

LLMs: Integrations with various language models (e.g., OpenAI, Hugging Face, Anthropic). These are the core engines that understand and generate human-like text.
Chat Models: Optimized for conversational interactions, often working with a list of messages (e.g., system, human, AI messages).
Embeddings: Converts text into numerical vectors, crucial for tasks like semantic search, retrieval, and similarity comparisons.
Prompts:

Prompt Templates: Standardized ways to construct prompts for LLMs. They allow you to dynamically insert variables into a predefined structure.
Example:

from langchain_core.prompts import PromptTemplate

prompt = PromptTemplate(template="What is a good name for a company that makes {product}?", input_variables=["product"])

print(prompt.format(product="colorful socks"))
# Output: What is a good name for a company that makes colorful socks?

  **Output Parsers:** Structure the output from LLMs into a desired format (e.g., JSON, Pydantic objects, lists). This helps make LLM outputs more usable programmatically.

     Example (Pydantic Output Parser):

 # W.I.P

Chains:

  * Chains are sequences of calls to LLMs or other utilities. They allow you to combine multiple components into a single, cohesive workflow.
 **Simple Chains:** A direct call to an LLM with a prompt.

 **Sequential Chains:** Execute a series of chains in order, passing the output of one as the input to the next.
 **Retrieval Chains:** Combine LLMs with retrieval models to fetch relevant documents and use them to inform the LLM's response (e.g., RAG - Retrieval Augmented Generation).

Retrieval:

  * **Document Loaders:** Load data from various sources (PDFs, websites, databases).
  * **Text Splitters:** Break down large documents into smaller, manageable chunks suitable for processing and embedding.
  * **Vector Stores:** Databases designed to store and efficiently query embeddings, enabling semantic search. Examples include FAISS, Chroma, Pinecone.
  * **Retrievers:** Interface to query a vector store and return relevant documents.

Agents:

  * Agents are a core concept in LangChain. They use an LLM as a "reasoning engine" to decide which **tools** to use and in what order, based on the user's input.
  * **Tools:** Functions that an agent can call to interact with the outside world (e.g., search engines, APIs, calculators, databases).
  * **Agent Executors:** The runtime that orchestrates the agent's actions, executing tools and feeding the results back to the LLM for further reasoning.
  * **Tool Calling:** A modern feature where LLMs can directly output structured calls to tools, often leveraging models fine-tuned for this purpose.

LangGraph: Building Statefule, Multi-Actor LLM Applications with Graphs

LangGraph extends LangChain by allowing you to build stateful, multi-actor applications as a directed acyclic graph (DAG) or even cyclic graphs (for agents with memory and self-correction). It's particularly useful for complex agentic workflows where you need to manage state, define execution paths, and enable features like human-in-the-loop or parallel execution.

Think of it this way: LangChain provides the building blocks; LangGraph provides the blueprint and orchestration engine to connect those blocks in sophisticated ways.

In LangGraph, an application is represented as a graph where:

Nodes: Represent a step or a component in your application (e.g., an LLM call, a tool invocation, a custom function).
Edges: Define the transitions between nodes, determining the flow of execution.
State: A shared object that is passed between nodes, allowing each node to read and update it. This is crucial for maintaining context and memory across multiple steps.

Key Components and Features of LangGraph:

State and StateSchema:

  * The **State** is the central concept in LangGraph. It's a mutable object that holds all the relevant information and context for the current execution of the graph. Each node receives the current state, performs its operation, and returns updates to the state.
  * `StateSchema`: This is a Pydantic model (or a dictionary) that defines the structure and types of the variables in your graph's state. It ensures type safety and helps manage complex state objects.

Nodes:

  * **Nodes** are the processing units of your graph. They can be any Python callable (function or method).
  * Each node takes the current graph state as input and returns a dictionary of updates to that state.

Edges:

  * **Edges** define the flow of control between nodes.
  * **Start/End Edges:** Specify where the graph execution begins and ends.
  * **Conditional Edges:** Allow for branching logic. Instead of directly specifying the next node, a conditional edge uses a function (a "router" or "decider") that takes the state and returns the name of the next node to execute. This is fundamental for agentic behavior where the LLM decides the next action.

  **Fixed Edges:** A direct, unconditional transition from one node to another.

HumanInLoop:

  * This feature allows for explicit human intervention at specific points in the graph. You can pause the execution, allow a human to review or modify the state, and then resume. This is crucial for validation, error correction, or supervised learning processes.
  * Implemented by designing a node that, when executed, might send a notification to a human and wait for an external signal to update the state or direct the flow. A common pattern is to have a conditional edge check a state variable like `awaiting_human_input`.

InMemory (State Persistence):

  * LangGraph's state is mutable and typically lives for the duration of a single `invoke` call. However, for continuous interactions or long-running agents, you often need to persist the state.
  * `InMemoryCheckpointSaver`: This is a simple way to store the graph's state in memory between executions. Useful for development and single-session interactions. For production, you'd integrate with databases (e.g., Redis, SQLite).

Streaming:

  * LangGraph supports streaming output from LLMs, allowing your application to display tokens as they are generated rather than waiting for the entire response. This improves user experience.
  * This is often managed by the underlying LLM integrations (e.g., `model.stream(messages)`) and how nodes are defined to yield intermediate results.

When invoking the graph, you would iterate over the stream() method: for s in graph_executor.stream(input): print(s).

Breakpoint:

  * Breakpoints allow you to pause the execution of the graph at specific nodes, inspect the current state, and potentially modify it before continuing. This is invaluable for debugging complex graph flows and understanding how the state evolves.
  * Enabled during graph compilation with `interrupt_before` or `interrupt_after` parameters.

Parallelization:

  * LangGraph can execute multiple nodes concurrently if they don't have dependencies on each other and operate on different parts of the state or perform independent computations. This can significantly speed up complex workflows.
  * This is achieved by defining parallel paths in your graph and using appropriate merging strategies for the state. You add multiple conditional routes from a node or define multiple `add_edge` or `add_conditional_edges` that might lead to nodes that run in parallel.

Subgraph:

  * You can compose graphs within graphs. A subgraph is essentially a self-contained LangGraph instance that can be treated as a single node within a larger graph. This promotes modularity and reusability, allowing you to build complex systems from smaller, manageable components.

MapReduce:

  * A common pattern in LLM applications, especially for summarization or data processing.
  * **Map:** A node processes multiple inputs (e.g., summarizes individual document chunks).
  * **Reduce:** Another node combines the outputs from the map phase (e.g., summarizes the individual summaries into a final summary).
  * LangGraph facilitates this by allowing you to define a node that iterates over a list in the state and then another node to aggregate the results.

ReAct (Reasoning and Acting):

  * A prominent agentic pattern where an LLM repeatedly performs a cycle of:
      * **Reason:** The LLM generates a thought/plan.
      * **Act:** The LLM decides which tool to use and its arguments, and the tool is invoked.
      * The observation from the tool is then fed back into the LLM for the next reasoning step.
  * LangGraph is an ideal framework for implementing ReAct agents due to its ability to model cycles and manage state across turns. A typical ReAct graph would have nodes for LLM reasoning, tool invocation, and conditional edges to loop back to reasoning until a final answer is determined. LangGraph's prebuilt `AgentExecutor` for `tool_calling` agents often uses a ReAct-like pattern under the hood.

Tool Calling:

  * Modern LLMs (like OpenAI's Function Calling or Anthropic's Tool Use) can directly output structured calls to tools in a machine-readable format.


In LangGraph, a node might receive the LLM's output, parse it to identify tool calls, invoke the specified tools, and then update the state with the tool's results, which can then be fed back to the LLM for further processing. This streamlines agent-tool interactions. LangGraph provides ToolExecutor and ToolNode classes to simplify this.

How LangGraph Connects to LangChain

LangGraph is built on top of LangChain. This means:

You use LangChain's components (LLMs, chat models, prompt templates, tools, retrievers) within your LangGraph nodes.
LangGraph provides the orchestration layer, connecting these LangChain components into a structured, stateful, and often cyclical workflow.

This combination allows developers to leverage the extensive toolkit of LangChain while gaining the fine-grained control and state management capabilities of LangGraph for building complex, robust, and interactive LLM applications.

Prompt Engineering, Techniques

Ashutosh Sarangi — Thu, 25 Sep 2025 06:29:56 +0000

🔧 Prompting Techniques with Templates

1. Zero-shot Prompting

Use when: The task is simple and doesn’t need examples.

🧪 Template:

Summarize the following email in one sentence: 
[Insert email text here]

🧠 Tip: Works best with clear, unambiguous instructions.

2. Few-shot Prompting

Use when: You want to guide the model with examples.

🧪 Template:

Convert the following sentences into passive voice:

Example 1:
Input: The cat chased the mouse.
Output: The mouse was chased by the cat.

Example 2:
Input: She writes a letter.
Output: A letter is written by her.

Now convert:
Input: They built a house.
Output:

🧠 Tip: Keep examples close in structure to your actual input.

3. Chain-of-Thought (CoT) Prompting

Use when: The task involves reasoning or multiple steps.

🧪 Template:

Question: A bookstore sells books for $12 each. If you buy 3 books and pay with a $50 bill, how much change do you get? 
Think step by step.

🧠 Tip: Add “Let’s think step by step” to encourage reasoning.

4. ReAct Prompting (Reasoning + Acting)

Use when: You want the model to reason and then take an action (e.g., search, call a tool).

🧪 Template (for agent-based systems):

Question: What is the capital of the country where the Eiffel Tower is located?

Thought: The Eiffel Tower is in Paris, which is in France. So the capital of France is Paris.

Action: Return "Paris"

🧠 Tip: Combine with LangGraph or LangChain agents for tool use.

5. Role Prompting

Use when: You want the model to adopt a specific persona or tone.

🧪 Template:

You are a senior backend engineer mentoring a junior developer. Explain the concept of RESTful APIs in simple terms with examples.

🧠 Tip: Great for tone control, teaching, or simulations.

6. Self-Consistency Prompting

Use when: You want more reliable answers from multiple generations.

🧪 Template:

Question: If a car travels 60 km in 1.5 hours, what is its average speed? Think step by step.

🧠 Tip: Run multiple generations and pick the most consistent answer.

7. Instruction + Format-Constrained Prompting

Use when: You need structured output (e.g., JSON, YAML, Markdown).

🧪 Template:

Extract the following email into structured JSON with keys: sender, subject, summary.

Email:
---
From: John Doe <john@example.com>
Subject: Meeting Reminder
Body: Just a reminder that we have a meeting tomorrow at 10 AM.
---

Output:
{
  "sender": "John Doe",
  "subject": "Meeting Reminder",
  "summary": "Reminder about a meeting scheduled for tomorrow at 10 AM."
}

🧠 Tip: Use this when integrating with APIs or downstream systems.

🧠 How to Avoid Hallucinations (with Prompting Tips)

Strategy	Prompting Tip
✅ Be explicit	“Only use facts from the following context:”
✅ Use RAG	“Based on the retrieved documents, answer the question.”
✅ Ask for sources	“Cite your sources for each fact.”
✅ Add verification	“Now review the above and point out any factual errors.”
✅ Lower temperature	Use `temperature=0.2` for factual tasks
✅ Avoid overloading	Keep prompts focused and relevant

Gen AI metadata

Ashutosh Sarangi — Thu, 25 Sep 2025 04:24:33 +0000

Basic

Introduction to Gen AI
- https://dev.to/ashutoshsarangi/introduction-to-gen-ai-1jdi
Gen AI, RAGS, CAG, Fine tuning, VectorDB, LangChain, LangSmith, LangGraph, Lang Flow, MCP, AI Agents, Agentic RAG
- https://dev.to/ashutoshsarangi/gen-ai-rags-cag-fine-tuning-vectordb-langchain-langsmith-langgraph-lang-flow-mcp-ai-51g5
Prompt Engineering, Techniques
- https://dev.to/ashutoshsarangi/prompt-engineering-techniques-1f73

Intermediate

Langchain and LangGraph
- https://dev.to/ashutoshsarangi/langchain-and-langgraph-1678

Advanced

From Localhost to Kubernetes: Containerizing an AI Log Analysis Agent
- https://dev.to/ashutoshsarangi/seamless-ai-agent-deployment-a-step-by-step-guide-with-real-world-challenges-516m

Advanced Python Concepts

Ashutosh Sarangi — Wed, 24 Sep 2025 05:49:00 +0000

Multithreading and Multiprocessing in Python

In Python, multithreading and multiprocessing are two ways to achieve concurrency, which is the ability to handle multiple tasks at the same time. While both can run multiple tasks, they do so in fundamentally different ways. The key difference lies in how they handle processes and threads.

Multithreading: In Python, multithreading involves a single process with multiple threads. Threads within the same process share the same memory space. This makes communication between threads very fast and efficient. However, Python's Global Interpreter Lock (GIL) restricts the execution of only one thread at a time on a single CPU core. This means multithreading is great for I/O-bound tasks ( network requests or file reads) but not for CPU-bound tasks (tasks that require heavy computation).
Multiprocessing: Multiprocessing, on the other hand, involves multiple independent processes, each with its own memory space and its own Python interpreter. Because they are separate processes, they can run on different CPU cores simultaneously, bypassing the GIL. This makes multiprocessing ideal for CPU-bound tasks that can be split into independent sub-tasks, as it can leverage multiple CPU cores to speed up execution.

Example: Multithreading vs. Multiprocessing

Let's use a simple example to illustrate the difference.

Multithreading Example (I/O-bound): A program that downloads multiple files from the internet. While one thread is waiting for a file to download, another can start downloading another file. This doesn't involve heavy computation, so multithreading is a good fit. The program's overall execution time is reduced because it's not waiting for one download to finish before starting the next.
Multiprocessing Example (CPU-bound): A program that performs complex mathematical calculations on a large dataset. By using multiprocessing, you can split the dataset and have a separate process perform calculations on each part. This allows the program to utilize multiple CPU cores, dramatically speeding up the total calculation time.

Synchronous vs. Asynchronous Programming

Synchronous programming is the default, sequential way of executing code. Each task must wait for the previous one to complete before it can begin. It's like a single-lane road where cars must follow one after another. If a task is slow, the entire program is blocked, and other tasks cannot proceed. This is simple and predictable but can be inefficient for tasks that involve waiting.

Asynchronous programming is a non-blocking approach that allows a program to initiate a task and then move on to other tasks without waiting for the first one to complete. It's like a chef taking an order, starting the dish, and then starting the next dish while the first one is cooking. When the first dish is ready, the chef can return to it. In Python, this is often implemented using the asyncio library, which uses an event loop to manage and schedule tasks.

Example: Sync vs. Async

Let's consider a web scraper that needs to fetch data from multiple websites.

Synchronous Example: The program fetches data from website A. It waits for the download to complete. Then it fetches data from website B. It waits for that download to complete, and so on. This process is sequential and can be very slow if one of the websites is slow to respond.
Asynchronous Example: The program initiates a request to website A, and while it's waiting for the response, it immediately sends a request to website B. It continues this process for all the websites. When a response from any of the websites arrives, the program can handle it. This approach is much more efficient because it uses the "waiting time" to do other work, allowing the program to fetch data from multiple sources concurrently without blocking.

Event loop in Python

In Python's asyncio, the event loop manages a single queue of tasks (coroutines) and callbacks. When a coroutine "awaits" an I/O operation (e.g., a network request), it yields control back to the event loop. The event loop then checks for completed I/O events and moves on to the next available task in its queue. The concept of "microtasks" is implicitly handled. When a task completes and schedules a callback (like the then part of an await operation), that callback is added directly to the event loop's queue and will be processed very soon.

There isn't a separate, higher-priority "micro" queue that gets fully drained before the main queue is checked again; instead, the asyncio loop is designed to be highly responsive to events and resume tasks as soon as they become ready.

Where Python's Event Loop is Located

Unlike JavaScript's event loop, which is an integral part of the browser or Node.js runtime, Python's event loop is part of the asyncio library itself. It's a key component of the asyncio module and is run as a single-threaded process within your Python application. When you use asyncio.run(), a new event loop is created, it runs all the tasks you've scheduled, and then it shuts down when they are complete.

The event loop is essentially a while True loop that continuously monitors and dispatches events. It's a central hub that:

Checks if any coroutines are ready to resume execution.
Handles completed I/O operations (e.g., from network sockets).
Manages scheduled callbacks and timers.

By default, there's only one event loop per thread, and it's generally recommended to run all your asyncio code within that one event loop in the main thread. To handle CPU-bound tasks without blocking the event loop, you would typically offload them to a separate thread or process using methods like loop.run_in_executor().

simple code examples to illustrate these concepts.

Multithreading vs. Multiprocessing

Multithreading is best for I/O-bound tasks (like downloading files or network requests) where the program spends most of its time waiting. The Global Interpreter Lock (GIL) limits multithreading to a single CPU core.

import threading
import time

def task(name):
    print(f"Thread {name}: Starting...")
    time.sleep(2)  # Simulate an I/O-bound task (e.g., waiting for a network response)
    print(f"Thread {name}: Finishing.")

threads = []
for i in range(3):
    t = threading.Thread(target=task, args=(i,))
    threads.append(t)
    t.start()

for t in threads:
    t.join() # Wait for all threads to complete

print("All threads have finished.")

In this example, three threads run concurrently. While one thread is sleeping (waiting), another can start, which is faster than running them sequentially.

Multiprocessing is ideal for CPU-bound tasks (like heavy mathematical calculations) because it bypasses the GIL by using separate processes, each with its own interpreter.

import multiprocessing
import time

def task(name):
    print(f"Process {name}: Starting...")
    time.sleep(2) # Simulate a CPU-bound task (e.g., complex calculation)
    print(f"Process {name}: Finishing.")

if __name__ == "__main__":
    processes = []
    for i in range(3):
        p = multiprocessing.Process(target=task, args=(i,))
        processes.append(p)
        p.start()

    for p in processes:
        p.join() # Wait for all processes to complete

    print("All processes have finished.")

Here, three independent processes are created. They can run on different CPU cores simultaneously, offering a true speed-up for CPU-intensive work.

Synchronous vs. Asynchronous Programming

Synchronous programming is sequential and blocking. Each task must wait for the previous one to complete.

import time

def fetch_data(url):
    print(f"Fetching data from {url}...")
    time.sleep(2) # Simulate a network request
    print(f"Finished fetching data from {url}.")
    return "Data from " + url

start_time = time.time()
data1 = fetch_data("website A")
data2 = fetch_data("website B")
data3 = fetch_data("website C")

print(f"All data fetched in {time.time() - start_time:.2f} seconds.")

The total time for this script to run will be approximately 6 seconds (2 seconds for each function call) because each call blocks the program until it finishes.

Asynchronous programming is non-blocking and event-driven. It allows the program to switch to other tasks while waiting for a long-running operation to complete. This is typically done with Python's asyncio library.

import asyncio
import time

async def fetch_data(url):
    print(f"Fetching data from {url}...")
    await asyncio.sleep(2) # Simulate a network request
    print(f"Finished fetching data from {url}.")
    return "Data from " + url

async def main():
    start_time = time.time()

    tasks = [
        fetch_data("website A"),
        fetch_data("website B"),
        fetch_data("website C")
    ]

    await asyncio.gather(*tasks) # Run tasks concurrently

    print(f"All data fetched in {time.time() - start_time:.2f} seconds.")

if __name__ == "__main__":
    asyncio.run(main())

In this asyncio example, the await asyncio.sleep(2) tells the program to pause this specific task but not block the entire program. Instead, the event loop can switch to another task. The total execution time will be around 2 seconds, as all three tasks are initiated and run concurrently. This is highly efficient for I/O-bound operations.

The primary benefit of asyncio is in managing concurrency, specifically for I/O-bound tasks. If you were making a hundred API calls, using asyncio would allow you to initiate all the requests concurrently, saving a significant amount of time by not waiting for each one to finish before starting the next. For a single call, there's no concurrency to manage, so the overhead of setting up an async function and an event loop provides no performance benefit.

Here’s a comparison:

Synchronous Approach (Recommended for a Single Call)

This approach is straightforward and easy to read. The code executes one line at a time.

import requests

def get_data_sync(url):
    response = requests.get(url)
    return response.json()

data = get_data_sync("https://api.example.com/data/1")
print(data)

Asynchronous Approach (Unnecessary for a Single Call)

While you can write a function with async and await, it's overkill for a single call. You need to use an async library like aiohttp and run the function within an event loop.

import asyncio
import aiohttp

async def get_data_async(url):
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as response:
            return await response.json()

async def main():
    data = await get_data_async("https://api.example.com/data/1")
    print(data)

if __name__ == "__main__":
    asyncio.run(main())

You don't put async and await in the get_data_sync function because it's a synchronous function, not an asynchronous one. It's using the requests library, which is a synchronous library.

Here's why:

requests library is synchronous: The requests.get() function is designed to be blocking. When you call it, your program stops and waits for the entire HTTP request to complete (sending the request, waiting for the server's response, and receiving the data) before it proceeds to the next line of code.
async and await require an asyncio ecosystem: The keywords async and await are part of Python's asyncio framework. They are used to define and manage coroutines, which are functions designed for non-blocking, asynchronous I/O operations. For these keywords to work, the function must be run within an event loop and use an asynchronous library like aiohttp that is compatible with asyncio.

Since your function uses requests, which is a synchronous library and doesn't communicate with an event loop, adding async and await would be incorrect and would lead to syntax errors or unexpected behavior.

Similarity between multithreading and `asyncio` in Python

Shared Goal: Concurrency in I/O-Bound Tasks

Both multithreading and asyncio are excellent for I/O-bound tasks. This is any task that spends most of its time waiting for an external operation to complete, like:

Making network requests: Waiting for a web server to respond.
Reading/writing files: Waiting for data to be read from or written to a disk.
Database queries: Waiting for a database to execute a query.

In these scenarios, the CPU is largely idle. Both multithreading and asyncio prevent your program from freezing while it waits, allowing it to work on other tasks in the meantime.

The Fundamental Difference: How They Handle Concurrency

The similarity ends at their core mechanism.

Multithreading uses preemptive multitasking. The operating system (OS) and the Python interpreter decide when to switch between threads. The threads themselves don't give up control voluntarily. This is what makes it "preemptive"—the OS preempts a running thread to give another one a turn. Because of Python's Global Interpreter Lock (GIL), only one thread can execute Python bytecode at a time, so it's not true parallelism but a form of interleaved concurrency.
asyncio uses cooperative multitasking. The tasks themselves (coroutines) voluntarily give up control to the event loop when they encounter an await keyword. The event loop is a single-threaded loop that checks for completed tasks and schedules the next one. This is what makes it "cooperative"—each task must cooperate by yielding control. Because it's single-threaded, it doesn't face the GIL limitations that multithreading does.

Metaprogramming in Python

It is the creation of programs that write or manipulate other programs. In essence, it's about making code that can inspect, generate, or modify itself at runtime. It’s a powerful technique often used to reduce code duplication, create flexible APIs, and build frameworks.

Key Concepts

Metaprogramming in Python is primarily achieved through:

Decorators: Functions that wrap other functions or classes to extend their behavior without permanent modification. They're a form of syntactic sugar for wrapping a callable.
Class Decorators: Similar to function decorators, but they modify the behavior of a class.
Metaclasses: The most advanced form of metaprogramming. A metaclass is the "class of a class." When you define a class, its metaclass is responsible for creating it. By creating a custom metaclass, you can control how classes are defined and how they behave.

Decorators in Python

A decorator in Python is a design pattern that allows you to dynamically extend or modify the behavior of a function or class without changing its source code. It's essentially a callable (like a function) that takes another callable as an argument and returns a new callable.

Decorators use a special syntax with the @ symbol, which is just syntactic sugar for a more verbose function call.

# This:
@my_decorator
def my_function():
    # ...

# Is equivalent to this:
def my_function():
    # ...

my_function = my_decorator(my_function)

The core idea is to "wrap" a function or class to add new functionality before or after the original code runs.

Function Decorators

A function decorator is a function that takes another function as an argument, adds some new functionality, and returns the modified function. This is most commonly used for tasks like logging, timing, authentication, or validation.

Example: A Simple Timer Decorator

This decorator measures how long a function takes to execute.

import time

def timing_decorator(func):
    """A decorator that prints the execution time of a function."""
    def wrapper(*args, **kwargs):
        start_time = time.time()
        result = func(*args, **kwargs)
        end_time = time.time()
        print(f"'{func.__name__}' ran in {end_time - start_time:.4f} seconds.")
        return result
    return wrapper

@timing_decorator
def complex_calculation(n):
    """Simulates a complex calculation."""
    sum_val = 0
    for i in range(n):
        sum_val += i
    return sum_val

# Calling the decorated function
result = complex_calculation(10000000)
print(f"Calculation result: {result}")

In this example, timing_decorator is the decorator function. It defines a new function wrapper that encapsulates the original complex_calculation function. When you call complex_calculation(10000000), you are actually calling the wrapper function. The wrapper executes its own timing logic, calls the original function, and then prints the elapsed time before returning the result.

Class Decorators

A class decorator is a callable that takes a class object as an argument and returns a new class object. Class decorators are used to modify or extend the behavior of an entire class. Common use cases include enforcing a class structure, adding attributes to all instances of a class, or automatically registering a class with a registry.

Example: A Class Decorator for Enforcing an Interface

This decorator checks if a class implements certain methods and raises an error if it doesn't.

def enforce_interface(cls):
    """A class decorator to ensure 'start' and 'stop' methods exist."""
    if not hasattr(cls, 'start') or not callable(getattr(cls, 'start')):
        raise TypeError(f"Class '{cls.__name__}' must have a 'start' method.")
    if not hasattr(cls, 'stop') or not callable(getattr(cls, 'stop')):
        raise TypeError(f"Class '{cls.__name__}' must have a 'stop' method.")
    return cls

@enforce_interface
class Car:
    def __init__(self, model):
        self.model = model

    def start(self):
        print(f"Starting the {self.model}.")

    def stop(self):
        print(f"Stopping the {self.model}.")

# This class will raise a TypeError when it's defined because it lacks a 'stop' method
try:
    @enforce_interface
    class Motorcycle:
        def __init__(self, model):
            self.model = model

        def start(self):
            print(f"Starting the {self.model}.")
except TypeError as e:
    print(f"\nCaught an error: {e}")

In this example, @enforce_interface is the class decorator. When the Car class is defined, the enforce_interface function is called with Car as an argument. The decorator checks for the required methods and returns the class. When Motorcycle is defined, the check fails, and the TypeError is raised immediately, providing static analysis at runtime.

Python Internals: A Deeper Look

Python is an interpreted, high-level, and dynamically typed language. Its inner workings, often referred to as "Python Internals," involve several key components that manage how your code is executed, including the CPython interpreter, the compilation process, memory management, and how core data types are handled. Understanding these can help you write more efficient and robust code.

The CPython Interpreter

When people talk about Python, they are usually referring to CPython, which is the reference implementation of the language written in C. It's the most widely used interpreter and is what you get when you download Python from python.org. Its main job is to take your Python source code and translate it into a language the computer can understand.

The execution of a Python script by the CPython interpreter follows these general steps:

Lexing and Parsing: The interpreter's front end reads your .py file. It first tokenizes the source code into a stream of tokens (lexing), such as keywords, identifiers, and operators. These tokens are then structured into an Abstract Syntax Tree (AST) , which represents the code's grammatical structure.
Compilation to Bytecode: The AST is then compiled into Python bytecode. Bytecode is a low-level, platform-independent set of instructions. It's not machine code but is a set of instructions for the Python Virtual Machine (PVM). You can see this bytecode using the dis module in Python.
Execution by the Python Virtual Machine (PVM): The PVM is the runtime engine of the CPython interpreter. It's a loop that reads and executes the bytecode instructions one by one. The PVM is also responsible for managing the call stack, memory, and objects.

The Global Interpreter Lock (GIL)

The Global Interpreter Lock (GIL) is one of the most talked-about aspects of CPython. It's a mutex (a lock) that protects access to Python objects, ensuring that only one thread can execute Python bytecode at a time. This simplifies memory management and prevents race conditions but means that multithreading in CPython cannot achieve true parallelism on multi-core processors for CPU-bound tasks.

Impact on I/O-Bound Tasks: The GIL is released during I/O operations (e.g., waiting for network data or disk reads). This allows other threads to run while the first is blocked, making multithreading still very effective for I/O-bound tasks.
Impact on CPU-Bound Tasks: For tasks that require heavy computation, the GIL prevents multiple threads from running simultaneously, negating the benefits of multi-core CPUs. In these cases, multiprocessing is the preferred approach, as each process has its own GIL.

Memory Management

Python's memory management is handled automatically. It uses a private heap to store objects and data structures. This heap is managed by the Python memory manager.

Reference Counting: The primary memory management strategy is reference counting. Each object has a counter that tracks the number of references pointing to it. When the reference count drops to zero, the object is deallocated, and its memory is returned to the heap.
Garbage Collection: Reference counting alone cannot handle circular references, where objects refer to each other but are no longer accessible from the main program. To solve this, Python's garbage collector periodically scans for these cycles and cleans up the orphaned objects.

Python Objects and Core Data Types

Everything in Python is an object, including integers, strings, functions, and even types themselves. Each object has:

PyObject_HEAD: A header that contains the reference count and a pointer to the object's type.
Type Pointer: A pointer to the object's type. For example, an integer object's type pointer would point to the PyInt_Type object.
Value: The actual data stored in the object.

Due to this object-oriented nature, even simple operations can be more complex than in low-level languages. For example, when you reassign a variable, you aren't changing the value in memory but rather changing the reference to point to a new object.

When you run a `.py` file

When you run a .py file, Python doesn't directly compile it into machine code like C or C++. Instead, it follows a multi-step process. Here is a step-by-step breakdown of how a Python program is compiled and how memory is allocated.

Step 1: Lexing and Parsing (The Frontend)

The CPython interpreter first reads your .py source file.

Lexing: The source code is broken down into a series of small, meaningful units called tokens. For example, a line of code like x = 10 + y would be broken into tokens for x, =, 10, +, and y.
Parsing: These tokens are then structured into an Abstract Syntax Tree (AST). The AST is a hierarchical tree representation of your code that reflects its grammatical structure and is easier for the interpreter to work with.

Step 2: Compilation to Bytecode

The AST is then passed to the compiler, which translates it into Python bytecode.

Bytecode is a low-level, platform-independent set of instructions. It's not machine code, but rather an instruction set for the Python Virtual Machine (PVM).
The .py file is not a binary executable; it's the source code. The compiled bytecode is usually saved in a .pyc file (Python compiled file) or the __pycache__ directory. This is done to speed up future executions of the same file, as the compilation step can be skipped.

Step 3: Execution by the Python Virtual Machine (PVM)

The PVM is the runtime engine of the Python interpreter. It is a loop that reads and executes the bytecode instructions one by one.

The PVM manages the program's call stack, which keeps track of active function calls.
It is also responsible for managing the program's memory.

Step 4: Memory Allocation and Object Creation

Python manages memory automatically using a private heap. All Python objects, including integers, strings, lists, and functions, are stored in this heap. When a variable is assigned to an object, two things happen:

Object Creation: An object is created in memory (on the heap) to store the data. The object's memory is allocated by the Python memory manager.
Reference Creation: A variable name is created in the current scope. This variable is a reference (a pointer) that points to the object in the heap.

Example:
When you run x = 10, the following happens:

The PVM sees the instruction to create an integer object with the value 10.
The memory manager finds a space in the heap and allocates memory for a new integer object.
A reference named x is created in the current scope, and it's set to point to the newly created object.

If you then run y = x, a new object is not created. Instead, a new reference y is created, and it also points to the same integer object 10.

Memory Deallocation

Python's memory management also handles deallocation automatically, primarily through reference counting.

Every object has a reference count that tracks how many variables are pointing to it.
When the reference count of an object drops to zero (e.g., when a variable goes out of scope or is reassigned), the memory manager deallocates the object and returns the memory to the heap.
Python also has a garbage collector to handle circular references (e.g., two objects that point to each other) that reference counting alone can't detect.

Sure, Ashutosh! Let's dive deep into context managers and generators in Python — two powerful features that help write clean, efficient, and readable code.

🧠 1. Context Managers

🔍 What is a Context Manager?

A context manager is a Python construct that allows you to allocate and release resources precisely when you want. The most common use case is with file operations, where you want to ensure a file is closed after you're done with it.

✅ Benefits:

Automatic resource management
Cleaner syntax using with statement
Exception-safe

🔧 Built-in Example:

with open('example.txt', 'r') as file:
    content = file.read()
print(content)  # File is automatically closed after the block

🛠 Custom Context Manager using `enter` and `exit`:

class MyContext:
    def __enter__(self):
        print("Entering context")
        return "Resource"

    def __exit__(self, exc_type, exc_value, traceback):
        print("Exiting context")

# Usage
with MyContext() as resource:
    print(f"Using {resource}")

🧹 Cleaner Custom Context Manager using `contextlib`:

from contextlib import contextmanager

@contextmanager
def managed_resource():
    print("Setup")
    yield "Resource"
    print("Cleanup")

with managed_resource() as res:
    print(f"Using {res}")

🔁 2. Generators

🔍 What is a Generator?

A generator is a function that returns an iterator and allows you to iterate over data without storing the entire sequence in memory. It uses the yield keyword.

✅ Benefits:

Memory-efficient
Lazy evaluation (generates values on the fly)
Useful for large datasets or streams

🔧 Simple Generator Example:

def count_up_to(max):
    count = 1
    while count <= max:
        yield count
        count += 1

for num in count_up_to(5):
    print(num)

🧠 Generator Expression (like list comprehension):

squares = (x*x for x in range(5))
for s in squares:
    print(s)

🔗 Combining Context Managers and Generators

You can use generators inside context managers or vice versa. For example, reading large files line-by-line:

def read_large_file(file_name):
    with open(file_name, 'r') as f:
        for line in f:
            yield line.strip()

for line in read_large_file('bigfile.txt'):
    print(line)

The with keyword in Python is specifically tied to context management. It is not a general-purpose OOP feature like class, def, or self. You can only use with on objects that implement the context management protocol, i.e., they define:

__enter__(self)
__exit__(self, exc_type, exc_value, traceback)

🔍 What Does `with` Actually Do?

When you write:

with managed_resource() as res:
    # do something with res

Python does the following under the hood:

Calls managed_resource().__enter__() and assigns its return value to res.
Executes the block under the with statement.
When the block finishes (even if an exception occurs), it calls __exit__() to clean up.

🧪 Can You Use `with` on Any Class?

Not directly. You must implement the context manager protocol in your class. Here's an example:

class MyClass:
    def __enter__(self):
        print("Entering context")
        return self

    def __exit__(self, exc_type, exc_value, traceback):
        print("Exiting context")

    def do_something(self):
        print("Doing something")

# Now you can use it with `with`
with MyClass() as obj:
    obj.do_something()

If your class doesn't implement __enter__ and __exit__, using with will raise a TypeError.

🧹 Shortcut: `contextlib.contextmanager`

If you don’t want to write a full class, you can use the @contextmanager decorator from contextlib to turn a generator function into a context manager:

from contextlib import contextmanager

@contextmanager
def simple_context():
    print("Setup")
    yield "Resource"
    print("Cleanup")

with simple_context() as res:
    print(f"Using {res}")

Writing Unit Tests for Python Code (unittest & doctest)

Ashutosh Sarangi — Tue, 23 Sep 2025 04:48:11 +0000

In Python, there are several tools to help you write, organize, run, and automate your unit tests. In the Python standard library, you’ll find two of these tools:

doctest
unittest (inspired by Junit)

Python’s doctest module is a lightweight testing framework that provides quick and straightforward test automation. It can read the test cases from your project’s documentation and your code’s docstrings.

The unittest package is also a testing framework. However, it provides a more complete solution than doctest.

unittest

Test case: An individual unit of testing. It examines the output for a given input set.
Test suite: A collection of test cases, test suites, or both. They’re grouped and executed as a whole.
Test fixture: A group of actions required to set up an environment for testing. It also includes the teardown processes after the tests run.
Test runner: A component that handles the execution of tests and communicates the results to the user.

# age.py

def categorize_by_age(age):
    if 0 <= age <= 9:
        return "Child"
    elif 9 < age <= 18:
        return "Adolescent"
    elif 18 < age <= 65:
        return "Adult"
    elif 65 < age <= 150:
        return "Golden age"
    else:
        return f"Invalid age: {age}"

# test_age.py

import unittest

from age import categorize_by_age

class TestCategorizeByAge(unittest.TestCase):
    def test_child(self):
        self.assertEqual(categorize_by_age(5), "Child")

    def test_adolescent(self):
        self.assertEqual(categorize_by_age(15), "Adolescent")

    def test_adult(self):
        self.assertEqual(categorize_by_age(30), "Adult")

    def test_golden_age(self):
        self.assertEqual(categorize_by_age(70), "Golden age")

    def test_negative_age(self):
        self.assertEqual(categorize_by_age(-1), "Invalid age: -1")

    def test_too_old(self):
        self.assertEqual(categorize_by_age(151), "Invalid age: 151")

Running unittest Tests

Make the test module executable
Use the command-line interface of unittest

1. Make the test module executable

# test_age.py

#...

if __name__ == "__main__":
    unittest.main()


# now you can directly run this file
# python test_age.py

Among other arguments, the main() function takes the verbosity one. With this argument, you can tweak the output’s verbosity, which has three possible values:

0 for quiet
1 for normal
2 for detailed
Go ahead and update the call to main() as in the following:

# ...

if __name__ == "__main__":
    unittest.main(verbosity=2)

# output

.
.
.
python test_age.py
test_adolescent (__main__.TestCategorizeByAge.test_adolescent) ... ok
test_adult (__main__.TestCategorizeByAge.test_adult) ... ok

If you want to make the detailed output more descriptive, then you can add docstrings to your tests like in the following code snippet:


# ...

class TestCategorizeByAge(unittest.TestCase):
    def test_child(self):
        """Test for 'Child'"""
        self.assertEqual(categorize_by_age(5), "Child")

    def test_adolescent(self):
        """Test for 'Adolescent'"""
        self.assertEqual(categorize_by_age(15), "Adolescent")

    def test_adult(self):
        """Test for 'Adult'"""
        self.assertEqual(categorize_by_age(30), "Adult")

# ...

# output
python test_age.py
test_adolescent (__main__.TestCategorizeByAge.test_adolescent)
Test for 'Adolescent' ... ok
test_adult (__main__.TestCategorizeByAge.test_adult)
Test for 'Adult' ... ok
test_boundary_adolescent_adult (__main__.TestCategorizeByAge.test_boundary_adolescent_adult)
Test for boundary between 'Adolescent' and 'Adult' ... ok

Skipping Tests

import sys
import unittest

class SkipTestExample(unittest.TestCase):
    @unittest.skip("Unconditionally skipped test")
    def test_unimportant(self):
        self.fail("The test should be skipped")

    @unittest.skipIf(sys.version_info < (3, 12), "Requires Python >= 3.12")
    def test_using_calendar_constants(self):
        import calendar

        self.assertEqual(calendar.Month(10), calendar.OCTOBER)

    @unittest.skipUnless(sys.platform.startswith("win"), "Requires Windows")
    def test_windows_support(self):
        from ctypes import WinDLL, windll

        self.assertIsInstance(windll.kernel32, WinDLL)

if __name__ == "__main__":
    unittest.main(verbosity=2)

Why do we need Sub tests?

class TestIsEven(unittest.TestCase):
    def test_even_number(self):
        for number in [2, 4, 6, -8, -10, -12]:
            self.assertEqual(is_even(number), True)

This will work fine, and the test will fail if any of the assertions fail. However, using subTest provides better granularity and diagnostics when running tests. Here's why it's useful:

✅ Benefits of `subTest`

Isolated Failures: Each iteration is treated as a separate subtest. If one number fails, the others still run, and you get a report for each.
Clearer Output: The test runner shows which specific input caused the failure, making debugging easier.
Improved Test Reporting: Especially useful when testing multiple inputs or edge cases.

Without `subTest`

If one assertion fails, the loop stops, and you don’t get feedback on the remaining values.

Example Output Comparison

With subTest:

FAIL: test_even_number (number=-10)
FAIL: test_even_number (number=-12)

Without subTest:

FAIL: test_even_number

Available Assert Methods

Comparing Values
Comparing the result of a code unit with the expected value is a common way to check whether the unit works okay. The TestCase class defines a rich set of methods that allows you to do this type of check:

Method Comparison
.assertEqual(a, b) a == b
.assertNotEqual(a, b) a != b
.assertTrue(x) bool(x) is True
.assertFalse(x) bool(x) is False

Comparing Objects by Their Identity
TestCase also implements methods that are related to the identity of objects. In Python’s CPython implementation, an object’s identity is the memory address where the object lives. This identity is a unique identifier that distinguishes one object from another.

An object’s identity is a read-only property, which means that you can’t change an object’s identity once you’ve created the object. To check an object’s identity, you’ll use the is and is not operators.

Here are a few assert methods that help you check for an object’s identity:

Method Comparison
.assertIs(a, b) a is b
.assertIsNot(a, b) a is not b
.assertIsNone(x) x is None
.assertIsNotNone(x) x is not None

Comparing Collections
Another common need when writing tests is to compare collections, such as lists, tuples, strings, dictionaries, and sets. The TestCase class also has shortcut methods for these types of comparisons. Here’s a summary of those methods:

Method Comparison
.assertSequenceEqual(a, b) Equality of two sequences
.assertMultiLineEqual(a, b) Equality of two strings
.assertListEqual(a, b) Equality of two lists
.assertTupleEqual(a, b) Equality of two tuples
.assertDictEqual(a, b) Equality of two dictionaries
.assertSetEqual(a, b) Equality of two sets

Using unittest From the Command Line

unittest test_age.py

you can also add modules to test

Grouping Your Tests With the TestSuite Class

import unittest

from calculations import (
    add,
    divide,
    mean,
    median,
    mode,
    multiply,
    subtract,
)

class TestArithmeticOperations(unittest.TestCase):
    def test_add(self):
        self.assertEqual(add(10, 5), 15)
        self.assertEqual(add(-1, 1), 0)

    def test_subtract(self):
        self.assertEqual(subtract(10, 5), 5)
        self.assertEqual(subtract(-1, 1), -2)

    def test_multiply(self):
        self.assertEqual(multiply(10, 5), 50)
        self.assertEqual(multiply(-1, 1), -1)

    def test_divide(self):
        self.assertEqual(divide(10, 5), 2)
        self.assertEqual(divide(-1, 1), -1)
        with self.assertRaises(ZeroDivisionError):
            divide(10, 0)

class TestStatisticalOperations(unittest.TestCase):
    def test_mean(self):
        self.assertEqual(mean([1, 2, 3, 4, 5, 6]), 3.5)

    def test_median_odd(self):
        self.assertEqual(median([1, 3, 3, 6, 7, 8, 9]), 6)

    def test_median_even(self):
        self.assertEqual(median([1, 2, 3, 4, 5, 6, 8, 9]), 4.5)

    def test_median_unsorted(self):
        self.assertEqual(median([7, 1, 3, 3, 2, 6]), 3)

    def test_mode_single(self):
        self.assertEqual(mode([1, 2, 2, 3, 4, 4, 4, 5]), [4])

    def test_mode_multiple(self):
        self.assertEqual(set(mode([1, 1, 2, 3, 4, 4, 5, 5])), {1, 4, 5})

if __name__ == "__main__":
    unittest.main()

These tests work as expected. Suppose you need a way to run the arithmetic and statistical tests separately. In this case, you can create test suites. In the following sections, you’ll learn how to do that.

# ...

# in the same way, we can make a suite for statistical

def make_suite():
    arithmetic_tests = [
        TestArithmeticOperations("test_add"),
        TestArithmeticOperations("test_subtract"),
        TestArithmeticOperations("test_multiply"),
        TestArithmeticOperations("test_divide"),
    ]
    return unittest.TestSuite(tests=arithmetic_tests)

if __name__ == "__main__":
    suite = make_suite()
    runner = unittest.TextTestRunner(verbosity=2)
    runner.run(suite)
# output

-> python test_calculations.py
test_add (__main__.TestArithmeticOperations.test_add) ... ok
test_subtract (__main__.TestArithmeticOperations.test_subtract) ... ok
test_multiply (__main__.TestArithmeticOperations.test_multiply) ... ok

Creating Test Fixtures

Great question! Let's break this down into two parts:

🔹 `setUp()`, `tearDown()`, `setUpClass()`, `tearDownClass()` in `unittest`

These are test fixture methods used to prepare and clean up the environment for your tests.

1. `setUp()`

Runs before each test method.
Used to set up objects or state needed for the test.

2. `tearDown()`

Runs after each test method.
Used to clean up resources (e.g., closing files, stopping mocks).

3. `setUpClass()`

Runs once before all tests in the class.
Must be decorated with @classmethod.

4. `tearDownClass()`

Runs once after all tests in the class.
Also decorated with @classmethod.

✅ Example:

import unittest

class MyTestCase(unittest.TestCase):

    @classmethod
    def setUpClass(cls):
        print("🔧 setUpClass: Run once before all tests")

    @classmethod
    def tearDownClass(cls):
        print("🧹 tearDownClass: Run once after all tests")

    def setUp(self):
        print("🔧 setUp: Run before each test")

    def tearDown(self):
        print("🧹 tearDown: Run after each test")

    def test_one(self):
        print("✅ Running test_one")

    def test_two(self):
        print("✅ Running test_two")

Output:

🔧 setUpClass: Run once before all tests
🔧 setUp: Run before each test
✅ Running test_one
🧹 tearDown: Run after each test
🔧 setUp: Run before each test
✅ Running test_two
🧹 tearDown: Run after each test
🧹 tearDownClass: Run once after all tests

🔹 What is `MagicMock`?

MagicMock is a subclass of Mock from unittest.mock. It behaves like a mock object but also includes default implementations for magic methods (like __len__, __getitem__, __iter__, etc.).

✅ Use Case:

from unittest.mock import MagicMock

mock_obj = MagicMock()
mock_obj.__len__.return_value = 5

print(len(mock_obj))  # Output: 5

🔸 Why Use `MagicMock`?

When you're mocking objects that use magic methods (dunder methods).
It saves you from manually defining those methods.

In Python unit testing, test fixtures are used to set up the environment before each test and clean it up afterward. When it comes to mocking instance methods and class methods, the unittest.mock module provides powerful tools to do this.

Here’s how you can mock both:

🔹 Mocking an Instance Method

Suppose you have a class like this:

class MyClass:
    def greet(self):
        return "Hello"

You can mock the greet method like this:

from unittest import TestCase
from unittest.mock import patch

class TestMyClass(TestCase):
    @patch.object(MyClass, 'greet', return_value='Mocked Hello')
    def test_greet(self, mock_greet):
        obj = MyClass()
        result = obj.greet()
        self.assertEqual(result, 'Mocked Hello')
        mock_greet.assert_called_once()

🔹 Mocking a Class Method

class MyClass:
    @classmethod
    def get_name(cls):
        return "Original Name"

You can mock it similarly:

from unittest import TestCase
from unittest.mock import patch

class TestMyClass(TestCase):
    @patch.object(MyClass, 'get_name', return_value='Mocked Name')
    def test_get_name(self, mock_get_name):
        result = MyClass.get_name()
        self.assertEqual(result, 'Mocked Name')
        mock_get_name.assert_called_once()

🔸 Key Points

Use @patch.object(ClassName, 'method_name') to mock methods.
For instance methods, you usually call them on an object.
For class methods, you call them on the class itself.
You can also use patch() as a context manager if you prefer not to use decorators.

✅ Bonus: Mocking with Fixtures

If you want to reuse mocks across multiple tests, you can use setUp():

class TestMyClass(TestCase):
    def setUp(self):
        patcher = patch.object(MyClass, 'greet', return_value='Hello from setup')
        self.mock_greet = patcher.start()
        self.addCleanup(patcher.stop)

    def test_greet(self):
        obj = MyClass()
        self.assertEqual(obj.greet(), 'Hello from setup')

Scenario Based Question

import datetime

def is_weekday():
    today = datetime.date.today()
    return 0 <= today.weekday() < 5
import weekday

class TestWeekday(unittest.TestCase):
    @patch("weekday.datetime")
    def test_is_weekday(self, mock_datetime):
        mock_datetime.date.today.return_value = datetime.date(2024, 4, 4)
        self.assertTrue(weekday.is_weekday())

    @patch("weekday.datetime")
    def test_is_weekend(self, mock_datetime):
        mock_datetime.date.today.return_value = datetime.date(2024, 4, 6)
        self.assertFalse(weekday.is_weekday())

Great questions! Let's break down both parts clearly:

✅ 1. Why are we mocking `datetime`?

You're mocking datetime.date.today() because:

The function is_weekday() depends on the current date.
In unit testing, you want predictable, repeatable results.
If you don’t mock it, the test result will change depending on the actual day it's run.

🔍 Without mocking:

def test_is_weekday():
    assert is_weekday()  # Might pass or fail depending on today's date

✅ With mocking:

@patch("weekday.datetime")
def test_is_weekday(self, mock_datetime):
    mock_datetime.date.today.return_value = datetime.date(2024, 4, 4)  # Thursday
    self.assertTrue(weekday.is_weekday())

This ensures the test always runs with a known date (April 4, 2024), which is a Thursday (weekday).

✅ 2. Who attaches the second parameter `mock_datetime` to the test method?

That’s done by the @patch decorator from unittest.mock.

How it works:

@patch("weekday.datetime") replaces datetime in the weekday module with a mock object.
It automatically injects that mock object as the first argument to your test method.
So mock_datetime is the mock version of datetime inside the weekday module.

Behind the scenes:

@patch("weekday.datetime")
def test_is_weekday(self, mock_datetime):  # mock_datetime is injected here

You can name it anything (mock_dt, mocked_datetime, etc.) — it’s just a parameter that receives the mock.

✅ Why `@patch("weekday.datetime")` instead of `@patch("datetime.datetime")`?

🔹 The key idea: Patch where it's used, not where it's defined.

When you write:

@patch("weekday.datetime")

You're patching the datetime as imported and used inside the weekday module — not the global datetime module.

🔍 Why this matters:

If you patch datetime.datetime globally, it affects all uses of datetime, which can lead to unintended side effects across your tests or even other modules.

But by patching weekday.datetime, you're only mocking the datetime object as seen by the weekday module — keeping the scope clean and safe.

✅ Alternative: Mock the whole function

You're absolutely right — another clean and often better approach is to mock the function directly, especially if it's small and self-contained.

Example:

Instead of mocking datetime, you could do:

@patch("weekday.is_weekday", return_value=True)
def test_is_weekday(self, mock_is_weekday):
    self.assertTrue(weekday.is_weekday())

This avoids mocking internals and focuses on behavior — which is often more robust and readable.

doctest: Document and Test Your Code at Once

🔹 What is `doctest`?

doctest lets you write tests inside your docstrings. It parses the docstring, runs the code snippets, and checks if the output matches.

✅ Example:

def add(a, b):
    """
    Adds two numbers.

    >>> add(2, 3)
    5
    >>> add(-1, 1)
    0
    """
    return a + b

Run with:

python -m doctest your_file.py

🔹 What is `unittest`?

unittest is Python’s built-in testing framework, inspired by Java’s JUnit. It supports:

Test fixtures (setUp, tearDown)
Assertions
Test discovery
Mocking
Parameterized tests

✅ Example:

import unittest

class TestAdd(unittest.TestCase):
    def test_add_positive(self):
        self.assertEqual(add(2, 3), 5)

    def test_add_negative(self):
        self.assertEqual(add(-1, 1), 0)

🔸 Comparison Table

Feature	`doctest`	`unittest`
Ease of use	✅ Very simple	❌ More boilerplate
Location of tests	Inside docstrings	Separate test classes/files
Readability	✅ Looks like examples	❌ More verbose
Flexibility	❌ Limited	✅ Very flexible
Mocking support	❌ Not supported	✅ Full support via `unittest.mock`
Fixtures	❌ Not supported	✅ `setUp`, `tearDown`, etc.
Best for	Simple functions, documentation	Complex logic, full test suites
Debugging	❌ Harder to debug failures	✅ Detailed error messages

✅ When to Use Each

Use `doctest` when:

You want to document and test simple functions at once.
You're writing educational or example-driven code.
You want lightweight testing with minimal setup.

Use `unittest` when:

You need fixtures, mocking, or parameterized tests.
You're testing complex logic, classes, or external dependencies.
You want structured, maintainable test suites.

🔸 Your Insight: Mocking and Fixtures in unittest

@patch and MagicMock for mocking dependencies.
setUp, tearDown, setUpClass for reusable test setup.
Integration with tools like pytest, coverage, and CI pipelines.

Reference:-

Behavioral interview Preparation

Ashutosh Sarangi — Fri, 19 Sep 2025 18:40:15 +0000

1. Challenges I faced

Scenario 1: Handling massive datasets without impacting browser performance

The Challenge: On a Catalogone project, we encountered a significant performance issue. We were building a feature that required loading a very long list of items, and while it worked fine with a small amount of data, the browser would slow down and eventually freeze when we had to display over a million elements. The core problem was trying to render all the elements in the DOM simultaneously.
What I Did: I took the initiative to solve this performance bottleneck by creating a custom npm package. The solution leveraged the IntersectionObserver API to monitor the visibility of a top and bottom element in the list. This allowed me to implement a virtualized list strategy: when a user scrolled down, the observer would trigger a function to add new elements to the bottom and remove older ones from the top. The inverse would happen when a user scrolled up.
The Outcome: My package ensured that at any given time, there were no more than 50 elements in the DOM, regardless of the total data size. This eliminated browser freezing and provided a smooth, seamless scrolling experience. The solution not only solved the immediate problem but also created a reusable, scalable package that could be used on other projects with large datasets, preventing similar issues in the future.

Scenario 2: Handling a tight deadline on a complex project

The Challenge: During the development of the "Phoenix" B2B product at Amdocs, we faced a tight deadline to launch a new module that needed to generate complex JSON objects for other applications. We discovered a critical bug in the conflict resolution feature that was causing data corruption just days before the scheduled release.
What I Did: I immediately initiated a rapid troubleshooting session with the team. Instead of panicking, I broke down the problem into smaller, manageable parts. I assigned specific tasks to team members based on their expertise, and I personally focused on debugging the core logic of the conflict resolution algorithm. I also communicated transparently with the project manager about the issue and our revised timeline for a fix, managing stakeholder expectations.
The Outcome: Through a focused and collaborative effort, we identified the root cause of the bug within a day. We implemented a fix, thoroughly tested it, and deployed the module with only a minimal delay. This experience highlighted the importance of clear communication and a calm, methodical approach under pressure.

2. Mistakes I made in the past

Scenario 1: Communication breakdown with a large team

The Mistake: As a Team Lead at Neosoft, I was responsible for a large-scale project involving a team of over 50 members. We were working on a complex feature that required tight coordination between different sub-teams (e.g., frontend, backend, mobile). My mistake was not establishing a formal communication channel for the team. I relied too much on informal communication channels, assuming everyone was on the same page. This led to a significant miscommunication where one sub-team built a feature based on an outdated set of API specs(Different JSON schema for response), causing a major integration roadblock and delaying the project.
What I Learned: This experience taught me a crucial lesson about the scalability of communication. With a small team, informal communication can work, but with a large team, a lack of formal processes can lead to chaos. I realized that clear, documented communication is a leader's most powerful tool for ensuring alignment and efficiency.
How I Rectified It: I immediately called a meeting with all the sub-team leads to address the issue. We collectively implemented a new communication protocol. We also introduced a brief daily sync-up meeting for team leads to flag any dependencies or potential roadblocks proactively. This new process streamlined our workflow and prevented similar issues on future projects.

Scenario 2: Underestimating the complexity of a new feature

The Mistake: While working on the "Catalog One" project, I was tasked with estimating the timeline for developing the "ABAC-Enable for Product Offering" (Attribute-Based Access Control) module. The task was to create a feature that could restrict/hide elements based on the rule and role. Based on my initial assessment, I provided an aggressive timeline to the epic. However, I underestimated the complexity of implementing the ABAC foundation, which caused us to fall behind schedule.
What I Learned: I learned to be more conservative with my timelines and to break down complex tasks into smaller, more detailed sub-tasks before providing an estimate. I also realized the importance of anticipating potential technical roadblocks and communicating them proactively.
How I Rectified It: I owned my mistake and immediately informed the product owners of the revised timeline, providing a detailed breakdown of the remaining work and the technical challenges we encountered. This transparency helped manage their expectations and allowed us to deliver a high-quality product, even if it was a bit later than originally planned.

3. I enjoyed in a project

Scenario 1: Building a multi-agent system

What I Enjoyed: I absolutely loved working on the "Multi-Agent-Toolcalling" project. The concept of designing a system where multiple LLMs (Large Language Models) could autonomously determine and coordinate tool calls was incredibly exciting. The project was at the cutting edge of AI.
Why It was Enjoyable: It was a unique blend of front-end development (designing the UI for displaying the agents' responses) and back-end logic (developing the system that orchestrates the tool calls). The problem-solving aspect was particularly rewarding, as it felt like I was building a truly intelligent system that could reason and make decisions. I also enjoyed seeing the results of our work in real-time, as the agents responded and performed tasks.

Scenario 2: The challenge of a real-time system

What I Enjoyed: I really enjoyed the "Human-in-Loop GenAI Integration" project. The real-time aspect of it (Socket.IO), was a thrilling challenge. It required a deep understanding of asynchronous communication and state management to ensure seamless interaction between the human user and the AI.
Why It was Enjoyable: The project was not just about coding; it was about designing a system that felt responsive and intuitive. It was a perfect blend of my front-end expertise with GenAI. Building the containerized environment with Docker and Kubernetes also gave me hands-on experience with scalable, modern infrastructure, which was a new and exciting area for me. The process of seeing the system come to life and observing the seamless interaction was incredibly satisfying.

4. Leadership role in projects

Scenario 1: Mentoring and building a large team

My Role: As a Team Lead at Neosoft, one of my most significant achievements was mentoring and developing a team of more than 50 members. My leadership wasn't just about managing tasks; it was about fostering growth and creating a strong, cohesive unit.
What I Did: I implemented a mentorship program where senior developers were paired with junior members. I also introduced weekly knowledge-sharing sessions where team members could present on a new technology or a challenging problem they solved. I focused on empowering my team by delegating responsibilities and trusting their abilities.
The Outcome: This approach not only improved the team's overall skill level and productivity but also boosted morale and reduced turnover. I'm proud that many of the developers I mentored went on to take on leadership roles themselves.

Scenario 2: Leading pre-sales and client interactions

My Role: As a Team Lead, I was responsible for leading many projects from the pre-sales stage all the way to completion. This included direct interaction with clients for requirement gathering and solution architecting.
What I Did: During pre-sales activities, I would meet with potential clients to understand their needs and propose a technical solution. For a project involving a web and mobile application, I listened to the client's business goals and then architected a full-stack solution using React for the front-end and Node.js for the back-end and we will wrap the application in Electron for mobile application. I created a detailed technical proposal and a proof of concept (POC) to demonstrate the feasibility of our solution.
The Outcome: My ability to translate business requirements into a clear technical roadmap and my hands-on experience with the technologies gave the client confidence in our team. We won the project and successfully delivered a solution that not only met their initial needs but also exceeded their expectations in terms of performance and scalability.

5. Conflicts in projects with colleagues

Scenario 1: Disagreement on a technical approach

The Conflict: On the "KnowledgeOne" project, a colleague and I had a disagreement about the best way to integrate Confluence with the LLM. I proposed using a RAG (Retrieval-Augmented Generation) system with Pinecone as the vector database, as I believed it was the most efficient and scalable solution for our embedded data storage. My colleague, however, preferred a different approach using a simpler, more traditional database, arguing it would be faster to implement.
How I Handled It: I listened carefully to my colleague's reasoning and acknowledged their point about the faster implementation time. However, instead of simply dismissing their idea, I presented a more detailed analysis of the long-term benefits of the RAG system, highlighting its scalability and the improved accuracy it would provide by retrieving information from our large Confluence knowledge base. I also offered to build a small prototype of my proposed solution to demonstrate its effectiveness.
The Outcome: My colleague appreciated the data-driven approach and the effort to build the prototype. After seeing the clear benefits of the RAG system, they agreed it was the better long-term solution. We moved forward with my proposed architecture, and the project was a success.

Scenario 2: The classic "backend vs. frontend" conflict

The Conflict: During the development of the "Phoenix" B2B product, there was a miscommunication between the frontend and backend teams regarding the format of a JSON object. The backend team had changed the structure of the JSON object without informing us on the frontend team, which caused our module to fail. My front-end colleagues were frustrated, and we were behind schedule.
How I Handled It: As a senior developer, I took the initiative to bridge the communication gap. I scheduled a quick meeting with the Lead and the relevant developers. I calmly explained the issue and suggested that we create a Confluence page where all changes to the API and JSON structures would be logged and communicated in real-time. This would prevent similar issues in the future.
The Outcome: The backend team was receptive to the idea. We implemented a new process of using a collaborative tool to document all API changes, which significantly improved our team's communication and efficiency. This experience reinforced the importance of proactive communication and process improvement in a collaborative environment.

Here is a different scenario for the same question, this time drawing from your experience as a Team Lead at Neosoft.

Scenario 3: Team Member with a highly creative problem-solver who preferred to "just get it done," often bypassing our standard documentation and code review processes

The challenge undocumented code caused integration issues for other developers on the project. My task was to address this without impacting their creative energy, which was a huge asset to the team.

I handled this by first scheduling a one-on-one meeting to understand their perspective.
I acknowledged their incredible speed and skill, recognizing that my personality might be too focused on process for their liking. Then, I explained the "why" behind our team standards: with a team of our size, clean code and clear documentation were not just a preference but a necessity for long-term project health and maintainability.

6. What I did differently apart from my org roles

Scenario 1: Self-learning and skill expansion

The Action: I have a strong passion for continuous learning that goes beyond my daily responsibilities. While working as a front-end developer, I made the conscious decision to expand my expertise into Generative AI and machine learning. This wasn't a requirement of my job at the time; it was a personal goal.
How it was Different: I dedicated my personal time to studying neural networks, lang graph, lang chain. I enrolled in online courses, read research papers, and worked on personal projects to get hands-on experience. This self-driven learning allowed me to become proficient in a new domain.
The Outcome: This independent learning proved to be a major advantage. It directly led to me being able to contribute to innovative projects like the "Human-in-Loop GenAI Integration" and the "Multi-Agent-Toolcalling" system at Amdocs, which were outside the scope of a typical front-end developer's role.

Scenario 2: Developing side projects to explore new technologies

The Action: I've always been a strong believer in building personal projects to apply what I learn. While working on enterprise applications, I decided to build an npm package that will handle a true infinite loader, without freezing the browser.
How it was Different: I faced lot of challenges to work on the details of the npm package, I read many articles, and I went very deep into how browser trace interaction works, which has more priority than all. But eventually, I did the npm package.
The Outcome: I have the npm package published, and I used in one of our internal projects. It worked very well.

Q. What keyword did your senior and junior colleges use for you?

Key Takeaway for Your Answer

What Juniors Would Say 🧑‍💻

Keywords: Helpful, Patient, Mentor, Go-to Person, Knowledgeable
Example Answer: "I believe my junior colleagues would describe me as the 'go-to person' on the team. They would say I'm patient and always willing to help. For example, a few months ago, a new developer on my team was struggling with a complex scenario with a thunk action in Redux. Instead of just giving them the solution, I sat with them, explained the underlying principles of the thunk action, and walked them through the debugging process. The goal wasn't just to fix the bug, but to help them understand the 'why' behind the code. I believe they would describe me as a mentor who invests in their growth."

What Senior People Would Say 👩‍💼

Keywords: Reliable, Strategic, Problem-Solver, Proactive, Owner
Example Answer: "My senior colleagues and managers would likely describe me as 'strategic' and a 'problem-solver'. They would say that I am someone who doesn't just write code, but also thinks about the long-term impact on the product and the business.
For instance, on our last project, I noticed a recurring performance issue caused by inefficient data fetching on the front-end. It wasn't my assigned task, but I proactively investigated it. In the UI component, it was being re-rendered multiple times. I added React.Memo and useCallbacks for optimising the re-renders. This improved our page load times by over 30%. They would describe me as an 'owner' who takes responsibility for the success of the entire feature."

Introduce Yourself

I'm a passionate and results-driven Front-End Developer with 10 years of experience building high-performance, scalable applications. My core expertise lies in modern technologies like React, Redux Toolkit, and TypeScript, which I've used to deliver complex projects throughout my career.
In my current role at Amdocs, I am an Advanced Software Developer working on an enterprise catalog project.
A key challenge I successfully solved was optimizing a massive data list, where I developed a custom npm package that prevented browser freezing by dynamically rendering only a handful of elements at any given time, ensuring a smooth user experience.
Before that, as a Team Lead at Neosoft, I not only worked hands-on with a wide range of technologies like React, React-Native, and Node.js, Ionic, Angular, but I also mentored and grew a team of over 50 members. I was directly involved in pre-sales activities and client interactions, where I'd architect and lead end-to-end project solutions.
I'm now expanding my skills into Generative AI and machine learning. My recent projects, like the Multi-Agent-Toolcalling and KnowledgeOne systems, allowed me to apply this knowledge by leveraging multiple LLMs and RAG systems.

I am now seeking a fast-paced and innovative environment where I can leverage my deep front-end expertise and growing knowledge in AI to build the next generation of applications, like yours.

Rebecca Okamoto Version of Introduce yourself

I'm a front-end expert with a decade of experience in building high-performance, scalable applications using React, Redux Toolkit, and TypeScript. My passion is tackling complex challenges and creating robust solutions that directly impact the user experience and business efficiency.

During my time as a Team Lead at Neosoft, I managed and mentored a team of over 50 members. I'm skilled at architecting end-to-end solutions, and I've successfully managed stakeholder expectations and pre-sales activities to ensure project success.

Finally, I'm at the forefront of innovation, expanding my expertise into Generative AI and machine learning. I have hands-on experience with transformer techniques and RAG systems, as demonstrated by my work on the Multi-Agent-Toolcalling and KnowledgeOne projects. I'm looking to bring this unique blend of front-end mastery and cutting-edge AI knowledge to a company like yours, where I can help build the next generation of applications.

Weakness:-

The Weakness: A Tendency to Over-Mentor and Provide Answers Too Quickly

1. The Weakness
"An earlier I've had to manage is my tendency to jump in and provide solutions for my team members too quickly. As a Team Lead at Neosoft, I was so focused on mentorship and ensuring project velocity that I would often solve a junior developer's problem for them, rather than guiding them to the answer themselves."

2. The Example
"I came to realize that while this helped in the short term, it wasn't scalable, especially with a team of over 50 members. More importantly, it was hindering my team's growth and critical thinking skills. They were becoming dependent on me as a single point of failure instead of learning to troubleshoot independently."

3. The Action & Growth

"To address this, I've made a conscious effort to change my approach. Now, when a team member comes to me with a problem, I shift from giving them the answer to asking them a series of probing questions. I'll guide them through the debugging process and encourage them to look at the documentation or code themselves.
I also implemented a knowledge-sharing system so the team could document their solutions for future reference.
This shift has not only empowered my colleagues to become more confident and independent problem-solvers but has also freed me up to focus on higher-level architectural challenges and strategic planning."

Python Basic init , pycache & PIP

Ashutosh Sarangi — Fri, 19 Sep 2025 09:04:48 +0000

Basic Python

1. Python Type Checking (Guide)

Python is dynamically typed, but you can use type hints for better readability, tooling, and static analysis.

✅ Type Hinting Example

def greet(name: str, age: int) -> str:
    return f"Hello {name}, you are {age} years old."

✅ Type Hinting for Complex Structures

Python uses the typing module to define complex types.

🔹 Imports you'll need:

from typing import Dict, List, Tuple, Any

🔸 Example 1: Function that returns a list of dictionaries

from typing import Dict, List

def transform_data(data: Dict[str, int]) -> List[Dict[str, str]]:
    """
    Converts a dict of {key: int} to a list of dicts with stringified values.

    Example:
    {"a": 1, "b": 2} → [{"key": "a", "value": "1"}, {"key": "b", "value": "2"}]
    """
    return [{"key": k, "value": str(v)} for k, v in data.items()]

✅ Type Breakdown

Dict[str, int]: Input is a dictionary with string keys and integer values.
List[Dict[str, str]]: Output is a list of dictionaries with string keys and string values.

🔸 Example 2: Function that returns a list of tuples

from typing import Dict, List, Tuple

def dict_to_tuples(data: Dict[str, int]) -> List[Tuple[str, int]]:
    """
    Converts a dict to a list of key-value tuples.

    Example:
    {"a": 1, "b": 2} → [("a", 1), ("b", 2)]
    """
    return list(data.items())

✅ Type Breakdown

List[Tuple[str, int]]: Output is a list of tuples, each containing a string and an integer.

🔸 Example 3: Mixed or Unknown Types

If your dictionary or list contains mixed types, use Any:

from typing import Dict, List, Any

def process(data: Dict[str, Any]) -> List[Dict[str, Any]]:
    return [{"key": k, "value": v} for k, v in data.items()]

🔹 Bonus: Using `TypedDict` for Structured Dicts

If your dicts have a fixed structure, you can define a TypedDict:

from typing import TypedDict, List

class User(TypedDict):
    name: str
    age: int

def get_users() -> List[User]:
    return [{"name": "Ashutosh", "age": 30}, {"name": "Assaf", "age": 40}]

Using TypedDict in Python provides several advantages, especially when you're working with dictionaries that have a fixed structure — like JSON-like data or configuration objects. Let's explore this in detail.

✅ What is `TypedDict`?

TypedDict is a feature from the typing module that lets you define the expected structure of a dictionary — including the types of its keys and values.

🔹 Example:

from typing import TypedDict

class User(TypedDict):
    name: str
    age: int
    is_active: bool

def get_user() -> User:
    return {"name": "Ashutosh", "age": 30, "is_active": True}

✅ Advantages of Using `TypedDict`

1. Type Safety

You get static type checking with tools like mypy, pyright, or IDEs (VS Code, PyCharm).

user: User = {"name": "Ashutosh", "age": "thirty", "is_active": True}  # ❌ Type error

2. Better IDE Support

Autocompletion for keys
Inline type hints
Error highlighting

3. Self-Documenting Code

It’s clear what structure the dictionary should have — no need to guess or read through comments.

def create_user(user: User) -> None:
    ...

4. Improved Refactoring

If you change the structure of User, type checkers will help you find all affected code.

5. Cleaner JSON Handling

When working with APIs, you often parse JSON into dictionaries. TypedDict helps define the expected shape.

import json

def parse_user(json_str: str) -> User:
    return json.loads(json_str)

🔸 Optional and Required Keys

You can define optional keys using total=False:

class PartialUser(TypedDict, total=False):
    name: str
    age: int

🔸 Comparison: `TypedDict` vs Raw `Dict[str, Any]`

Feature	`Dict[str, Any]`	`TypedDict`
Type safety	❌ No	✅ Yes
IDE support	❌ Limited	✅ Autocompletion, hints
Documentation	❌ Implicit	✅ Explicit structure
Error detection	❌ Runtime only	✅ Static analysis

✅ When Should You Use `TypedDict`?

When working with structured data (e.g., JSON from APIs)
When you want type-safe dictionaries
When building data models without needing full classes
When you want to avoid overengineering with dataclasses or pydantic

Absolutely! Let's dive into how to use TypedDict with required and optional keys in Python, with clear examples and explanations.

✅ What is `TypedDict`?

TypedDict allows you to define the structure of a dictionary with specific key names and value types. It’s part of the typing module and is especially useful when working with structured data like JSON.

🔹 Required vs Optional Keys

By default, all keys in a TypedDict are required. You can make keys optional by using total=False or by using NotRequired (Python 3.11+).

✅ Example 1: Required Keys Only

from typing import TypedDict

class User(TypedDict):
    name: str
    age: int

def get_user() -> User:
    return {"name": "Ashutosh", "age": 30}  # ✅ All keys are required

If you omit a key:

return {"name": "Ashutosh"}  # ❌ mypy will complain: 'age' is missing

✅ Example 2: Optional Keys with `total=False`

from typing import TypedDict

class PartialUser(TypedDict, total=False):
    name: str
    age: int
    email: str

def get_partial_user() -> PartialUser:
    return {"name": "Ashutosh"}  # ✅ Only 'name' is provided

Here, all keys are optional — you can include any, all, or none.

✅ Example 3: Mixed Required and Optional (Python 3.11+)

from typing import TypedDict, NotRequired

class UserProfile(TypedDict):
    username: str
    age: int
    email: NotRequired[str]
    phone: NotRequired[str]

def get_profile() -> UserProfile:
    return {
        "username": "ashu_dev",
        "age": 30,
        "email": "ashu@example.com"
    }  # ✅ 'phone' is optional

This is the most flexible and readable way to define mixed key requirements.

🔸 Summary

Method	Python Version	Behavior
`TypedDict`	3.8+	All keys required
`TypedDict, total=False`	3.8+	All keys optional
`NotRequired`	3.11+	Selectively optional keys

🧩 1. `BaseModel` (from `pydantic`)

✅ Features

Runtime data validation and type enforcement
Automatic serialization/deserialization (e.g., .json(), .dict())
Useful for APIs, config files, and anywhere you need robust data models
Supports default values, validators, and nested models

🧪 Example:

from pydantic import BaseModel, Field

class User(BaseModel):
    name: str
    age: int = Field(..., ge=0)  # age must be >= 0

🔍 Behavior:

If you pass invalid data, it raises a ValidationError.

User(name="Alice", age=-5)  # ❌ Raises error due to age < 0

🧩 2. `TypedDict` (from `typing` or `typing_extensions`)

✅ Features

Static type checking only (via tools like MyPy, Pyright)
No runtime validation
Lightweight and faster
Good for defining dict-like structures with known keys

🧪 Example:

from typing import TypedDict

class UserDict(TypedDict):
    name: str
    age: int

🔍 Behavior:

No runtime checks — it's just a hint for linters and IDEs.

user: UserDict = {"name": "Alice", "age": -5}  # ✅ No error at runtime

🆚 Summary Comparison

Feature	`BaseModel` (Pydantic)	`TypedDict` (typing)
Runtime validation	✅ Yes	❌ No
Type hints	✅ Yes	✅ Yes
Serialization	✅ `.dict()`, `.json()`	❌ Manual
Performance	❌ Slower	✅ Faster
Use case	APIs, configs, data models	Lightweight dicts
Nested models	✅ Supported	❌ Manual nesting

🧠 When to Use What?

Use BaseModel when:
- You need data validation
- You're building APIs (e.g., FastAPI)
- You want structured, reliable data
Use TypedDict when:
- You want lightweight type hints
- You don’t need runtime validation
- You're working with plain dicts and want IDE/type checker support

2. Duck Typing in Python

Duck Typing is a concept where the type of an object is determined by its behavior (methods/attributes), not its actual class.

“If it walks like a duck and quacks like a duck, it’s a duck.”

✅ Example

class Duck:
    def quack(self):
        print("Quack!")

class Person:
    def quack(self):
        print("I'm pretending to be a duck!")

def make_it_quack(thing):
    thing.quack()

make_it_quack(Duck())    # Quack!
make_it_quack(Person())  # I'm pretending to be a duck!

🔸 Benefits

Flexibility
Decoupled code
Easier testing and mocking

>>> numbers = [1, 2, 3]
>>> person = ("Jane", 25, "Python Dev")
>>> letters = "abc"
>>> ordinals = {"one": "first", "two": "second", "three": "third"}
>>> even_digits = {2, 4, 6, 8}
>>> collections = [numbers, person, letters, ordinals, even_digits]

>>> for collection in collections:
...     for value in collection:
...         print(value)
...
1
2
3
Jane
25
Python Dev
a
b
c
one
two
three
8
2
4
6

✅ General Operations on Built-in Collections

Operation	Lists	Tuples	Strings	Ranges	Dictionaries	Sets
Iteration	✅	✅	✅	✅	✅	✅
Indexing	✅	✅	✅	✅	❌	❌
Slicing	✅	✅	✅	✅	❌	❌
Concatenating	✅	✅	✅	❌	❌	❌
Finding length	✅	✅	✅	✅	✅	✅
Reversing	✅	✅	✅	✅	❌	❌
Sorting	✅	✅*	✅*	❌	❌	✅

* Sorting tuples and strings returns a new sorted list, not a sorted tuple/string.

🔹 Detailed Examples

✅ Iteration

for x in [1, 2, 3]: print(x)
for x in (1, 2, 3): print(x)
for x in "abc": print(x)
for x in range(3): print(x)
for k in {"a": 1, "b": 2}: print(k)
for x in {1, 2, 3}: print(x)

✅ Indexing

print([1, 2, 3][0])       # 1
print((1, 2, 3)[1])       # 2
print("hello"[2])         # 'l'
print(range(5)[3])        # 3
# Dictionaries and sets do not support indexing

✅ Slicing

print([1, 2, 3][1:])      # [2, 3]
print((1, 2, 3)[1:])      # (2, 3)
print("hello"[1:4])       # 'ell'
print(range(10)[2:5])     # range(2, 5)

✅ Concatenating

print([1, 2] + [3, 4])    # [1, 2, 3, 4]
print((1, 2) + (3, 4))    # (1, 2, 3, 4)
print("ab" + "cd")        # 'abcd'
# Sets use union, not +

✅ Finding Length

print(len([1, 2, 3]))
print(len((1, 2)))
print(len("hello"))
print(len(range(10)))
print(len({"a": 1, "b": 2}))
print(len({1, 2, 3}))

✅ Reversing

print(list(reversed([1, 2, 3])))
print(tuple(reversed((1, 2, 3))))
print("".join(reversed("abc")))
print(list(reversed(range(5))))
# dict and set are unordered → not reversible directly

✅ Sorting

print(sorted([3, 1, 2]))          # [1, 2, 3]
print(sorted((3, 1, 2)))          # [1, 2, 3]
print(sorted("cba"))             # ['a', 'b', 'c']
print(sorted({3, 1, 2}))          # [1, 2, 3]
# dicts can be sorted by keys or values manually

🔸 Notes

Dictionaries are unordered mappings — you can iterate over keys, values, or items, but not index or slice.
Sets are unordered collections — no indexing or slicing, but support union, intersection, etc.
Tuples and strings are immutable — operations like sorting or reversing return new objects.

3. Sets in Python

Sets are unordered collections of unique elements.

✅ Basic Usage

a = {1, 2, 3}
b = {3, 4, 5}

# Union
print(a | b)  # {1, 2, 3, 4, 5}

# Intersection
print(a & b)  # {3}

# Difference
print(a - b)  # {1, 2}

# Symmetric Difference
print(a ^ b)  # {1, 2, 4, 5}

🔸 Set Comparison

a == b       # Checks if sets are equal
a.issubset(b)
a.issuperset(b)

4. Other Data Types

✅ List

fruits = ["apple", "banana", "cherry"]
fruits.append("orange")

✅ Dictionary

person = {"name": "Ashutosh", "age": 30}
print(person["name"])

✅ Tuple

point = (10, 20)

✅ String

text = "Hello, World!"

5. Why Use Dictionary When We Have JSON?

🔹 JSON is a data format (text-based).

🔹 Dictionary is a Python data structure.

You use dictionaries to work with data in Python, and convert them to/from JSON when communicating externally (e.g., APIs, files).

✅ Example

import json

data = {"name": "Ashutosh", "age": 30}
json_str = json.dumps(data)  # Convert dict to JSON string
parsed = json.loads(json_str)  # Convert JSON string back to dict

6. `sorted()` vs `.sort()`

✅ `sorted()`

Returns a new sorted list
Works on any iterable
Doesn’t modify the original

nums = [3, 1, 2]
new_nums = sorted(nums)

✅ `.sort()`

Sorts the list in-place
Only works on lists
Returns None

nums = [3, 1, 2]
nums.sort()

🔸 Why Both?

sorted() is functional and flexible.
.sort() is efficient for large lists when mutation is acceptable.

✅ Summary

Concept	Key Takeaway
Type Checking	Improves safety and tooling
Duck Typing	Behavior > Type
Sets	Unique, unordered, fast ops
Lists, Dicts, Tuples	Core data structures
Dict vs JSON	Dict = Python, JSON = format
`sorted()` vs `.sort()`	New list vs in-place sort

Why we need init?

1. Purpose of init.py

init.py file in a directory tells Python that the directory should be treated as a package.
This allows you to import modules from that directory using package syntax, e.g.:
Without init.py, Python (pre-3.3) would not recognize the folder as a package, and imports might fail.
The file can be empty, or it can contain initialization code for the package.

Starting with Python 3.3 and above (including Python 3.11), you can technically remove the init.py file from a package directory, and Python will still recognize it as a package due to "implicit namespace packages."

However, you should keep init.py:

When you add an init.py file to a package directory, it does not create a global namespace. Instead, it defines a package namespace.
All modules and submodules inside that package share the package’s namespace (e.g., tools_package.audio.utils).
This keeps your package’s contents organized and separate from the global namespace, preventing naming conflicts.

Module vs Package

Module: A single Python file (e.g., utils.py). You import it like import utils or from utils import foo.
Package: A directory containing an init.py file and (usually) multiple modules (e.g., audio/ with init.py and utils.py).

2. What is pycache?

When you run or import Python code, Python compiles .py files to bytecode for faster execution.
The compiled bytecode files are stored in the pycache directory, with names like utils.cpython-311.pyc.
cpython-311 means the file was compiled by CPython version 3.11.
These .pyc files are used by Python to speed up future imports of the module.

3. PIP (pip installs packages) (package Manager for Python):-

Using pip in a Python Virtual Environment

$ python -m venv venv/
$ source venv/bin/activate

(venv) $ pip3 --version
pip 24.2 from .../python3.12/site-packages/pip (python 3.12)

(venv) $ pip --version
pip 24.2 from .../python3.12/site-packages/pip (python 3.12)

Here you initialize a virtual environment named venv by using Python’s built-in venv module.
After running the command above, Python creates a directory named venv/ in your current working directory.
Then, you activate the virtual environment with the source command. - The parentheses (()) surrounding your venv name indicate that you successfully activated the virtual environment.
Finally, you check the version of the pip3 and pip executables inside your activated virtual environment.
Both point to the same pip module, so once your virtual environment is activated, you can use either pip or pip3.

Installing Packages With pip

 install a package
       pip install package_name
 uninstall a package
       pip uninstall package_name
 list installed packages
       pip list
 install a specific version of a package
       pip install package_name==version_number
 upgrade a package
       pip install --upgrade package_name
 show package information
       pip show package_name
 search for packages
       pip search package_name

Using Requirements Files

 requirement.text package
       pip install -r requirements.txt
 make a requirements.txt file from the packages installed
       pip freeze > requirements.txt

certifi==x.y.z
charset-normalizer==x.y.z
idna==x.y.z
requests>=x.y.z, <3.0
urllib3==x.y.z

Changing the version specifier for the requests package ensures that any version greater than or equal to 3.0 doesn’t get installed.

very important

We can create 3 files

requirements_dev.txt (For Development)
requirements_prod.txt (For Production)
requirements_lock.txt (After Development complete --> We will freeze it)

Uninstalling Packages With pip

pip show <package Name>

Name: requests
Version: 2.32.3
Summary: Python HTTP for Humans.
Location: .../python3.12/site-packages
Requires: certifi, idna, charset-normalizer, urllib3
Required-by:

Notice the last two fields, Requires and Required-by. The show command tells you that requests requires certifi, idna, charset-normalizer, and urllib3. You probably want to uninstall those too. Notice that requests isn’t required by any other package.
So it’s safe to uninstall it.

pip uninstall certifi urllib3 -y

Here you uninstall urllib3. Using the -y switch, you suppress the confirmation dialog asking you if you want to uninstall this package.

In a single call, you can specify all the packages that you want to uninstall

To create a virtual environment in Python, you can use the built-in venv module. A virtual environment is a self-contained directory that contains a Python installation for a particular version of Python, plus a number of additional packages.

✅ Steps to Create and Use a Virtual Environment

🔧 1. Create the Virtual Environment

python -m venv myenv

python: Use python3 if you're on macOS/Linux and python points to Python 2.
myenv: This is the name of the virtual environment folder. You can name it anything.

📂 2. Activate the Virtual Environment

Windows:

myenv\Scripts\activate

macOS/Linux:

source myenv/bin/activate

Once activated, your terminal prompt will change to show the environment name, e.g., (myenv).

📦 3. Install Packages Inside the Virtual Environment

pip install requests

This installs requests only inside myenv, not globally.

❌ 4. Deactivate the Virtual Environment

deactivate

This returns you to the global Python environment.

Functional Programming In Python

Ashutosh Sarangi — Fri, 19 Sep 2025 05:15:54 +0000

What Is Functional Programming?

A pure function is a function whose output value follows solely from its input values without any observable side effects.
In functional programming, a program consists primarily of the evaluation of pure functions.
Computation proceeds by nested or composed function calls without changes to state or mutable data.

In Python, functions are first-class citizens.

This means that functions have the same characteristics as values like strings and numbers.
Anything you would expect to be able to do with a string or number, you can also do with a function.

>>> def func():
...     print("I am function func()!")
...

>>> func()
I am function func()!

>>> another_name = func
>>> another_name()
I am function func()!

**>>> def func():
...     print("I am function func()!")
...

>>> print("cat", func, 42)
cat <function func at 0x7f81b4d29bf8> 42**

function composition / Higher order functions

When you pass a function to another function, the passed-in function is sometimes referred to as a callback.
function can also specify another function as its return value

>>> def outer():
...     def inner():
...         print("I am function inner()!")
...     # Function outer() returns function inner()
...     return inner
...

>>> function = outer()
>>> function
<function outer.<locals>.inner at 0x7f18bc85faf0>
>>> function()
I am function inner()!

>>> outer()()
I am function inner()!

Closure and bag Pack

Python also supports closure and its bag pack concept (Lexical Persistent Reference value) L.P.R.V

def outer():
    x = 23
    def inner():
        print(x, ' Inner')
    return inner;


inner_fun = outer()


inner_fun()

Defining an Anonymous Function With lambda

Functional programming is all about calling functions and passing them around, so it naturally involves defining a lot of functions.
Sometimes, it’s convenient to be able to define an anonymous function on the fly without having to give it a name. In Python, you can do this with a lambda expression.

lambda <parameter_list>: <expression>
(lambda x1, x2, x3: (x1 + x2 + x3) / 3)(9, 6, 6) # Self-invoking fun
(lambda x: "even" if x % 2 == 0 else "odd")(2)

>>> reverse = lambda s: s[::-1]
>>> reverse("I am a string")
'gnirts a ma I'

Applying a Function to an Iterable With map()

def reverse(s):
    return s[::-1]

animals = ["cat", "dog", "hedgehog", "gecko"]
x = list(map(reverse, animals))

print(x)

Calling map() With Multiple Iterables

map(, , , ..., )



def add_three(a, b, c):
    return a + b + c


y = list(map(add_three, [1, 2, 3], [10, 20, 30], [100, 200, 300]))
print(y)

#[111,222,333] (1, 10, 100), (2, 20, 200)

Selecting Elements From an Iterable With filter()

>>> def greater_than_100(x):
...     return x > 100
...
>>> list(filter(greater_than_100, [1, 111, 2, 222, 3, 333]))
[111, 222, 333]

Reducing an Iterable to a Single Value With reduce()

reduce() applies a function to the items in an iterable two at a time, progressively combining them to produce a single result.

def f(x, y):
...     return x + y
...

>>> from functools import reduce
>>> reduce(f, [1, 2, 3, 4, 5])
# 15

Calling reduce() With an Initial Value

>>> def f(x, y):
...     return x + y
...

>>> from functools import reduce
>>> reduce(f, [1, 2, 3, 4, 5], 100)  # (100 + 1 + 2 + 3 + 4 + 5)
115

Python Metadata

Ashutosh Sarangi — Wed, 17 Sep 2025 13:18:40 +0000

Cheat Sheet

https://realpython.com/cheatsheets/python/#getting-started

Basic

Python Basic init , pycache & PIP
- https://dev.to/ashutoshsarangi/python-basic-init-pycache-pip-3dkl

Intermediate

Intermediate Python OOP For Gen AI
- https://dev.to/ashutoshsarangi/intermediate-python-oop-for-gen-ai-2lim
Functional Programming In Python
- https://dev.to/ashutoshsarangi/functional-programming-in-python-cka
Unit Testing
- https://dev.to/ashutoshsarangi/writing-unit-tests-for-python-code-unittest-doctest-iob

Advanced

Advanced Python Concepts
- https://dev.to/ashutoshsarangi/advanced-python-concepts-3ci8
Python Performance Optimization: Detailed Guide
- https://dev.to/ashutoshsarangi/python-performance-optimization-detailed-guide-5739

Specialization

will update soon
- link

Architecture

Folder Structure
- https://dev.to/ashutoshsarangi/python-project-folder-structure-for-a-genai-application-13kc

Python project folder structure for a GenAI application

Ashutosh Sarangi — Wed, 17 Sep 2025 11:42:00 +0000

📁 Project Structure Overview

genai_app/
│
├── app/
│   ├── __init__.py
│   ├── main.py               # FastAPI app entry point
│   ├── config.py             # Environment/config settings
│   ├── models/               # Pydantic models and data schemas
│   │   └── genai.py
│   ├── services/             # Core GenAI logic (e.g., LangChain, transformers)
│   │   └── genai_service.py
│   ├── api/                  # Route definitions
│   │   ├── __init__.py
│   │   └── genai_routes.py
│   ├── utils/                # Helper functions, logging, etc.
│   │   └── helpers.py
│   └── middleware/           # Custom middleware (e.g., logging, auth)
│       └── auth.py
│
├── tests/                    # Unit and integration tests
│   ├── __init__.py
│   └── test_genai.py
│
├── requirements.txt          # Python dependencies
├── .env                      # Environment variables
├── README.md                 # Project documentation
└── run.sh                    # Shell script to run the app with Uvicorn

🚀 Key Components Explained

`main.py`

from fastapi import FastAPI
from app.api.genai_routes import router as genai_router

app = FastAPI(title="GenAI FastAPI App")
app.include_router(genai_router)

`genai_routes.py`

from fastapi import APIRouter
from app.services.genai_service import generate_response
from app.models.genai import GenAIRequest

router = APIRouter()

@router.post("/generate")
def generate_text(request: GenAIRequest):
    return generate_response(request.prompt)

`genai_service.py`

def generate_response(prompt: str) -> dict:
    # Call to GenAI model (e.g., OpenAI, HuggingFace, LangChain)
    return {"response": f"Generated text for: {prompt}"}

🧪 Running the App

`run.sh`

#!/bin/bash
uvicorn app.main:app --reload --host 0.0.0.0 --port 8000

🧠 Optional Enhancements

Docker support: Add Dockerfile and docker-compose.yml
Async support: Use async def in routes and services
Model integration: Add HuggingFace or OpenAI SDK in genai_service.py
LangGraph or LangChain: Integrate in services/ for advanced workflows

Forem: Ashutosh Sarangi

(aiohttp & asyncio) vs Requests: Comparing Python HTTP Libraries

1. Requests - The Simple Synchronous Library

What it is:

When to use:

Example:

Pros:

Cons:

2. asyncio - The Foundation (Not an HTTP Library!)

What it is:

Key Concepts:

Example (without HTTP):

Why asyncio alone isn't enough for HTTP:

3. aiohttp - Async HTTP Client/Server

What it is:

When to use:

📊 Side-by-Side Comparison---

🔥 Real Performance Example

🔍 What exactly happens with session.get(url)?

Step 1: Understanding what fetch_one(url) returns

Step 2: What's in the tasks list?

Step 3: What does asyncio.gather() do?

* and ** Unpacking

* (Single asterisk) - Unpacking Iterables

** (Double asterisk) - Unpacking Dictionaries

Python Performance Optimization: Detailed Guide

**product Object.model_dump()

Python Performance Optimization: Detailed Guide

1. Overview: Optimize What Needs Optimizing

Why This Matters

The Right Approach

2. Sorting Optimization

❌ Avoid: Using Comparison Functions

✅ Use: key Parameter with operator.itemgetter

✅ Use: sorted() for Non-Destructive Sorting

Advanced Sorting Techniques

Decorate-Sort-Undecorate (DSU) Pattern (Legacy)

Sorting Stability (Important!)

Performance Comparison

When to Use What

3. String Concatenation

❌ Avoid: Using += for String Building

✅ Use: join() Method

Performance Comparison

String Formatting

3. Loops and Iteration

❌ Avoid: Manual Loop with Append

✅ Use: List Comprehensions

✅ Use: map() for Simple Operations

Map in detail

🧠 Syntax of map()

✅ Example 1: Using Built-in Function str.upper

✅ Example 2: Using a Lambda Function

✅ Example 3: Using a Custom Function

Generator Expressions (Memory Efficient)

Generator Expression in Detail

🧠 What Is a Generator?

1. Using a Generator Function with yield

2. Using a Generator Expression

🔄 What Is yield?

✅ Example: Using yield

Output:

🔁 What a Generator Does

🧠 Why the for Loop Is Needed

4. Avoiding Dots (Attribute Lookups)

❌ Avoid: Repeated Attribute Lookups

BAD - Looks up .append and .upper on every iteration

✅ Use: Cache Attribute Lookups

GOOD - Lookup once, use many times

Real-World Example

❌ Bad - repeated lookups

✅ Good - cache lookups

5. Local vs Global Variables

❌ Avoid: Global Variables in Loops

BAD - Accessing globals is slow

✅ Use: Local Variables

GOOD - Local variables use optimized storage

Best Practice

❌ Avoid

✅ Use

🔍 What exactly happens with `session.get(url)`?

Step 1: Understanding what `fetch_one(url)` returns

Step 3: What does `asyncio.gather()` do?

`*` and `**` Unpacking

`*` (Single asterisk) - Unpacking Iterables

`**` (Double asterisk) - Unpacking Dictionaries

🧠 Syntax of `map()`

✅ Example 1: Using Built-in Function `str.upper`

1. Using a Generator Function with `yield`

🔄 What Is `yield`?

✅ Example: Using `yield`

🧠 Why the `for` Loop Is Needed