Forem: Pragati Verma

[Boost]

Pragati Verma — Thu, 20 Mar 2025 13:09:48 +0000

Understanding Python's Garbage Collection and Memory Optimization

Pragati Verma ・ Mar 17

#programming #python #computerscience #codenewbie

Understanding Python's Garbage Collection and Memory Optimization

Pragati Verma — Mon, 17 Mar 2025 12:47:35 +0000

Memory management is one of the most important yet underestimated aspects of programming. Most developers focus on features and functionality, often neglecting this hidden foundation until problems arise. Poor memory management leads to programs that run slowly, crash without warning, or gradually consume more memory until they fail.

However, unlike other programming languages such as C, C++ and Java, where developers must allocate and deallocate memory explicitly, Python automates this process using Reference Counting and Garbage Collection.

In this article, we will discuss Python's memory management system in depth. We'll start with how Python organizes memory, then understand reference counting and its limitations, and finally discuss how garbage collection solves these problems.

Here's an overview:

How Python organizes memory (stack memory and heap memory)
Python's primary memory management: Reference counting
Limitations of reference counting
Python's garbage collection system
Practical strategies for memory optimization
Real-world applications and performance considerations

Before learning about how Python tracks and cleans up memory, it's essential to understand how memory is structured. Let's start with the fundamentals of how Python organizes memory.

Memory Organization in Python

To handle memory allocation effectively, Python organizes memory into two main regions:

Stack Memory: This stores function calls, local variables, and control flow. It is automatically managed, with memory being allocated and freed as functions execute.
Heap Memory: This is used for dynamically allocated objects such as lists, dictionaries, and user-defined instances. These objects remain in memory until they are no longer needed.

Stack Memory

Stack memory handles function calls, local variables, and references to objects. Each time a function is called, a new stack frame is created, storing:

Function arguments
Local variables (names referencing objects)
Control flow information

Once the function completes, the stack frame is discarded, and its references are removed.

For example:

def example():
    x = [10, 20, 30]  # 'x' is stored in stack memory
    return x

result = example()  # 'result' now references the returned list

Here, x exists in the stack memory during function execution, but the list [10, 20, 30] is created in heap memory. When the function returns, x is removed from the stack, but since result still references the list, it remains in heap memory.

Heap Memory

All dynamically allocated objects (lists, dictionaries, user-defined objects) are stored in heap memory. As long as references exist, these objects persist beyond function execution.

x = [5, 6, 7]  # The list is created in heap memory

In this example:

The list [5, 6, 7] resides in heap memory and the name x is stored in stack memory and acts as a reference to the heap object

Now that we understand how memory is organized let's examine how Python keeps track of these heap objects by counting references.

Python's Primary Memory Management: Reference Counting

In many programming languages, such as C, C++, and Java, variables and objects are distinct - variables directly store values or memory addresses pointing to objects. However, in Python, variables are merely names (labels) that reference objects in memory rather than storing values themselves. Python uses reference counting to track how many names point to a given object, as multiple names can reference the same object.

Each object in Python consists of three key attributes:

Type: The object's data type is inferred automatically by Python
Value: The actual data stored within the object
Reference Count: The number of references (or names) pointing to the object.

Python provides a sys.getrefcount() function that can be used to check an object's reference count:

import sys

a = [1, 2, 3]
print(sys.getrefcount(a))  # Output: 2 (one for 'a', one for function argument)

The count is 2 because calling sys.getrefcount(a) temporarily creates another reference as a function argument.

The reference count increases each time a new reference is created and decreases every time a reference is deleted or reassigned.

For example:

a = [1, 2, 3]  # Reference count: 1
b = a          # Reference count: 2 (both 'a' and 'b' reference the same list)
a = None       # Reference count: 1 (only 'b' references the list now)
del b          # Reference count: 0 (no references left)

When an object's reference count drops to zero, Python automatically deallocates the memory, making it available for future use. This automatic cleanup is a significant advantage of Python's memory management system.

Advantages of Reference Counting

Reference counting offers several benefits as a memory management strategy:

Immediate Cleanup – When an object's reference count reaches zero, it is deallocated instantly, preventing excessive memory consumption.
Simplicity – The mechanism is straightforward and does not require complex background processes.
Deterministic Behavior – Since objects are freed as soon as they become unreferenced, we can predict when memory will be released.

However, despite these advantages, reference counting alone is insufficient for comprehensive memory management. Let's examine its limitations.

Limitations of Reference Counting

While reference counting efficiently handles most memory management scenarios, it has several significant limitations:

Performance Overhead: Incrementing and decrementing reference counts adds a small but constant performance cost, especially in large applications.
Not Thread-Safe: Reference counting can cause issues in multi-threaded programs due to potential race conditions.
Circular References: The biggest limitation is that objects referencing each other keep their reference count above zero, preventing automatic deallocation.

The problem of circular references deserves special attention.

Consider this example:

class Node:
    def __init__(self):
        self.ref = None

a = Node()
b = Node()

a.ref = b  # 'a' references 'b'
b.ref = a  # 'b' references 'a'

del a
del b  # Objects still exist due to circular reference

In this code, even after deleting the variables a and b, the Node objects themselves continue to exist in memory because they reference each other, keeping their reference counts at 1 instead of 0. This creates a memory leak that reference counting alone cannot resolve.

Thus, the circular reference problem represents a fundamental limitation of reference counting that requires an additional mechanism to resolve. This is where Python's garbage collection system comes into play as a complementary solution to reference counting.

Python's Garbage Collection System

Garbage collection (GC) is Python's complementary memory management system that works alongside reference counting to clean up objects no longer in use, particularly those involved in circular references. It frees memory allocated to unused objects, preventing memory leaks.

Generational Garbage Collection

Python's GC follows a generational approach to optimize performance. The idea is that most objects have short lifespans, so collecting them frequently improves efficiency.

The garbage collector divides objects into three generations based on their age:

Generation 0 (Youngest): Newly created objects. Collected most frequently.
Generation 1 (Intermediate Age): Objects that survived a previous collection.
Generation 2 (Oldest): Long-lived objects, such as global variables and cached data.

How Garbage Collection Works Step by Step

1. Starts with Generation 0 (Youngest Objects)

When garbage collection runs, it first checks Generation 0, which contains the newest objects.
Since short-lived objects (e.g., temporary variables in functions) are most likely to become garbage quickly, this step helps free memory efficiently.

2. Objects That Survive Are Promoted

If an object is still referenced after a collection, it is moved to the next generation (Generation 1) instead of being deleted.
This means Python assumes that if an object has survived one garbage collection, it might be needed for a longer period.
Similarly, objects in Generation 1 that survive another collection are promoted to Generation 2.

3. Generation 2 (Long-Lived Objects) Is Collected Least Often

Since objects in Generation 2 have already survived multiple collections, Python assumes they are important and unlikely to be garbage.
As a result, Generation 2 is collected the least frequently, reducing performance overhead.
Examples of such objects include module-level variables, global caches, and persistent data structures.

This incremental collection prevents unnecessary performance overhead while keeping memory usage efficient.

Detecting and Breaking Circular References

Python's garbage collector periodically scans memory to find objects that mutually reference each other but are no longer accessible from the rest of the program.

Here's how it works with our previous example:

import gc

class Node:
    def __init__(self):
        self.ref = None

a = Node()
b = Node()
a.ref = b
b.ref = a

del a
del b  # Objects still exist due to circular reference

gc.collect()  # Manually trigger garbage collection

Here, calling gc.collect() forces Python to scan for circular references. The garbage collector identifies that the two Node objects form a reference cycle that's no longer accessible from the program and removes them, freeing up memory.

When Does Garbage Collection Run?

Python's garbage collector automatically runs in the background based on a threshold system:

A collection cycle is triggered if the number of object allocations exceeds a set limit.
Developers can manually trigger garbage collection using gc.collect(), though this is rarely needed.
The garbage collector can also be disabled with gc.disable() if performance optimization is required in specific cases.

While reference counting handles most memory deallocation immediately, the garbage collector provides a safety net that catches circular references and other complex memory issues that reference counting cannot address.

Performance Considerations

While Python's GC system is generally efficient, frequent garbage collection cycles can cause minor performance slowdowns, especially in memory-intensive applications.

We can also fine-tune collection behavior using the gc module:

Adjust thresholds using gc.set_threshold().
Monitor collection statistics with gc.get_stats().
Disable automatic collection in performance-critical sections and run it manually.

Now that we understand Python's memory management, let's look at practical strategies for optimizing memory usage in your applications.

Optimizing Memory Management in Python

While Python's garbage collection system handles memory cleanup automatically, unoptimized memory usage can lead to performance bottlenecks.

The following strategies help in managing memory more efficiently:

Avoiding Unnecessary Object Creation

Creating too many objects unnecessarily increases memory usage and strains Python's garbage collector. Using memory-efficient techniques, such as generators instead of lists, can significantly reduce memory consumption.

Example: Using Generators Instead of Lists

# List consumes memory for all elements at once
squares_list = [x * x for x in range(1000000)]  

# Generator computes values on demand, reducing memory usage
squares_generator = (x * x for x in range(1000000))

Since generators yield values one at a time, they prevent storing large lists in memory, making them ideal for iterating over large datasets.

Handling Circular References Efficiently

Circular references occur when objects reference each other, preventing reference counts from reaching zero. While Python's cyclic garbage collector detects and cleans up these objects, using weak references can help break reference cycles and optimize memory usage.

Example: Using Weak References to Prevent Cycles

import weakref

class Example:
    pass

obj = Example()
weak_ref = weakref.ref(obj)  # Creates a weak reference instead of a strong one

Unlike normal references, weak references do not increase the reference count. Once an object has no strong references left, it is automatically garbage-collected, even if weak references point to it.

Disabling or Controlling Garbage Collection When Necessary

In high-performance applications (such as real-time systems or machine learning workloads), automatic garbage collection may cause performance overhead. Disabling it temporarily and running manual collections at controlled intervals can help.

Example: Disabling Garbage Collection

import gc

gc.disable()  # Disable automatic garbage collection

# Perform high-performance operations...

gc.collect()  # Manually trigger garbage collection when needed
gc.enable()   # Re-enable automatic garbage collection

This approach is useful when frequent GC cycles disrupt performance-sensitive operations.

Monitoring Memory Usage

Tracking memory allocation helps identify leaks and optimize memory usage. Python provides tools like tracemalloc and objgraph to monitor memory usage effectively.

Example: Using `tracemalloc` to Track Memory Usage

import tracemalloc

tracemalloc.start()

# Code execution
print(tracemalloc.get_traced_memory())  # Shows current and peak memory usage

tracemalloc.stop()

By analyzing memory snapshots, developers can identify and optimise objects consuming excessive memory.

Real-World Applications

Understanding Python's memory management becomes particularly important in several scenarios:

Data Processing Applications

When working with large datasets, efficient memory management can be the difference between a functional application and one that crashes due to memory errors. Techniques like generators, chunking data, and processing in batches can significantly reduce the memory footprint.

Long-Running Services

Even small memory leaks can accumulate over time and cause failures for server applications and microservices that run continuously. Regular profiling and monitoring of memory usage help identify and fix these issues early.

Resource-Constrained Environments

In environments with limited memory (embedded systems, serverless functions with tight memory limits), understanding how Python manages memory allows developers to write more efficient code and avoid hitting resource limits.

Conclusion

Python's memory management system offers immediate cleanup through reference counting and comprehensive leak prevention through garbage collection. This dual approach ensures that memory is used efficiently with minimal manual intervention from developers.

However, relying solely on automatic memory management isn't always optimal. Understanding how Python handles memory allows us to write more efficient, high-performing applications.

By minimizing unnecessary object creation, using weak references, controlling garbage collection, and monitoring memory usage, we can prevent excessive memory consumption and improve application performance. Thoughtful memory management is not just about avoiding leaks but more about writing scalable, optimized code.

Next time when you debug a memory issue or optimize a Python application, remember how the underlying memory management works. These insights can help you make better design decisions and create more efficient software.

Setting Up Apache Kafka Locally: A Step-by-Step Guide

Pragati Verma — Tue, 21 Jan 2025 16:30:47 +0000

The previous article explored Apache Kafka’s key features, architecture and real-world applications.

Mastering Apache Kafka: Powering Modern Data Pipelines

Pragati Verma ・ Jan 16

#computerscience #webdev #database #beginners

In this article, we’ll walk through the steps to set up Apache Kafka on your local machine. By the end, you'll have a fully functional Kafka environment to start experimenting with data streams.

Prerequisites:

Before we start, make sure you have the following installed:

Java 8 or later (Kafka is built on Java)
Homebrew (macOS package manager)

Step 1: Install Kafka Using Homebrew

Homebrew simplifies the installation of Kafka and its dependencies. Open your terminal and run the following commands:

Install Zookeeper (required by Kafka):

brew install zookeeper

Install Kafka:

brew install kafka

This will install Kafka along with its dependencies, including Zookeeper.

Step 2: Start Zookeeper

Kafka depends on Zookeeper for distributed coordination. You need to start Zookeeper first.

In your terminal, run:

zkServer start

Leave this terminal window open, as Zookeeper will need to keep running.

Step 3: Start Kafka Server

Now that Zookeeper is running, let’s start the Kafka server.

In a new terminal window, run the following command:

kafka-server-start /usr/local/etc/kafka/server.properties

This starts the Kafka server on the default port (9092).

Step 4: Create a Kafka Topic

Kafka organizes data into topics. Let’s create a simple topic for testing.

Run the following command in a new terminal window:

kafka-topics --create --topic test --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

This command creates a topic called test with one partition and one replica.

Step 5: Produce Messages to the Topic

To send messages to the test topic, start a producer session. In a new terminal, run:

kafka-console-producer --broker-list localhost:9092 --topic test

Now, you can type messages and press Enter to send them to the test topic.

For example:

Hello, Kafka!

Step 6: Consume Messages from the Topic

Start a consumer session to see the messages you sent to the test topic. Open another terminal and run:

kafka-console-consumer --bootstrap-server localhost:9092 --topic test --from-beginning

You should see the messages appear as they are consumed from the test topic.

Hello, Kafka!

Step 7: Verify Everything is Running

Congrats! You’ve successfully set up Kafka and tested it locally on your macOS machine. You can now explore more Kafka features, such as Producers, Consumers, and Consumer Groups.

Step 8: Experiment Further

Send more messages through the producer and watch them appear in the consumer.

Start multiple consumers for the same topic to see how Kafka distributes messages.

Common Issues and Troubleshooting Tips

Setting up Apache Kafka can sometimes present challenges, especially for beginners. Here are a few common issues and troubleshooting tips to help ensure a smooth setup:

1. Kafka Broker Won't Start:

Error: If the Kafka broker fails to start and you see errors related to binding to the port (e.g., Port 9092 is already in use), this usually means that the default Kafka port is occupied by another process.
Solution:
- Check if any other process is using port 9092 by running netstat -an | find "9092" (Windows) or lsof -i :9092 (Linux/Mac).
- If so, either stop the conflicting process or change the Kafka port in the config/server.properties file by modifying the listeners property:

  listeners=PLAINTEXT://localhost:9093

2. Zookeeper Not Starting:

Error: When starting Zookeeper, you might see errors like Failed to bind to /0.0.0.0:2181.
Solution:
- Ensure that no other service is using port 2181, which is Zookeeper’s default port.
- Check that the Zookeeper data directory is correctly configured in zookeeper.properties. If the directory does not exist or is incorrect, create or specify the correct directory.
- Try clearing the Zookeeper data directory (zookeeper-data) and then restart Zookeeper.

3. Kafka Producer Cannot Connect to Broker:

Error: The producer might fail to connect to the broker, throwing errors like Broker not available or Connection refused.
Solution:
- Verify that Kafka is running by checking the logs. Look for entries like Started Kafka server to confirm it's operational.
- Ensure the bootstrap-server parameter in your producer command points to the correct broker and port. If you changed the default port, update it in your producer configuration.
- Make sure your Kafka server is accessible from the machine where you're running the producer. If you're running Kafka on a different machine, ensure that the network connection and firewall rules allow communication.

4. Kafka Consumer Not Receiving Messages:

Error: The consumer might not display any messages even though the producer is sending them.
Solution:
- Check that the consumer is subscribed to the correct topic. You can use the kafka-topics command to verify that the topic exists:

.\bin\windows\kafka-topics.bat --list --bootstrap-server localhost:9092

Ensure the consumer is consuming from the correct partition, especially in topics with multiple partitions.
If using multiple consumers, make sure they are properly consuming messages from the topic. Kafka consumers in the same consumer group will share message processing load. If there are fewer partitions than consumers, some consumers might not receive any messages.
Ensure the --from-beginning flag is used if you want the consumer to read from the start of the topic (not just the latest messages).

5. Topic Creation Failures:

Error: When trying to create a topic, you might see an error like The broker does not have a topic with the specified name.
Solution:
- Ensure that your Kafka broker is running correctly and that you have access to the specified bootstrap-server.
- Check if the topic already exists using the kafka-topics command:

.\bin\windows\kafka-topics.bat --describe --topic MyTopic --bootstrap-server localhost:9092

If needed, delete the topic and recreate it:

.\bin\windows\kafka-topics.bat --delete --topic MyTopic --bootstrap-server localhost:9092

6. Kafka Consumer Groups Not Working Properly:

Error: Kafka consumer groups may not function as expected, with some consumers not processing messages.
Solution:
- Ensure that each consumer in the group is reading from a different partition. If the topic has more partitions than consumers, the consumers will balance the load, but there may be idle consumers.
- Monitor consumer lag using the kafka-consumer-groups command:

.\bin\windows\kafka-consumer-groups.bat --describe --group <consumer-group-name> --bootstrap-server localhost:9092

7. Zookeeper and Kafka Version Mismatch:

Error: Version incompatibility issues between Kafka and Zookeeper may arise when using a newer Kafka version with an older version of Zookeeper or vice versa.
Solution:
- Make sure that both Zookeeper and Kafka are compatible. If you're using Kafka 2.8.0 or higher and planning to use KRaft mode, you don’t need Zookeeper anymore. Otherwise, ensure that both components are using compatible versions.

By following these troubleshooting steps and tips, most common Kafka setup issues can be resolved quickly, enabling smooth operation of the Kafka ecosystem.

Conclusion

Setting up Apache Kafka locally can seem challenging due to the various components involved, but with careful configuration and troubleshooting, it becomes manageable.

By understanding common errors and their resolutions, you can ensure a smooth setup process. Once Kafka is up and running, it unlocks a powerful platform for handling real-time data streams, enabling you to explore features like producers, consumers, and topic management.

This foundation prepares you to integrate Kafka into your projects and explore advanced use cases. With practice, you’ll master its potential for building scalable, distributed systems that handle massive data flows seamlessly.

Are you curious about how modern systems handle massive amounts of real-time data? Apache Kafka is the secret behind seamless data streams, from powering stock market analytics to real-time social media insights. Start your Kafka journey today!

Pragati Verma — Mon, 20 Jan 2025 14:09:06 +0000

Mastering Apache Kafka: Powering Modern Data Pipelines

Pragati Verma ・ Jan 16

#computerscience #webdev #database #beginners

Mastering Apache Kafka: Powering Modern Data Pipelines

Pragati Verma — Thu, 16 Jan 2025 12:33:50 +0000

Imagine Uber tracking millions of rides every minute, and each ride generates tons of data, such as driver location, passenger requests, and real-time traffic conditions. Traditional databases struggle to handle such a high-speed data influx, causing slowdowns, inconsistencies, and delays. This is where Kafka saves the day, enabling real-time data streaming that ensures seamless processing at scale.

In this article, we will get an in-depth understanding of Apache Kafka’s architecture, its key components, and the reasons why it is indispensable for real-time data streaming. By the end, you will gain insights into Kafka’s powerful capabilities and how it addresses challenges that traditional databases are often ill-equipped to handle.

So, let's dive right in!

What is Apache Kafka?

Kafka was originally developed by LinkedIn in 2010 to address the challenges it faced with its growing data pipeline. The need for a reliable, high-throughput messaging system that could handle real-time data led to the creation of Kafka. In 2011, it was open-sourced, and since then, it has grown in popularity, becoming a central component in many organizations' data infrastructures.

It is an open-source, distributed event streaming platform that handles large volumes of real-time data. It allows applications to publish, subscribe to, store, and process streams of records, making it ideal for scenarios that require high-speed data handling and real-time analytics.

Apache Kafka was named after Franz Kafka, the Czech writer known for exploring complex and overwhelming systems in his works. The name reflects Kafka's design to handle massive, intricate data streams, like navigating chaotic systems in Kafka's literature, offering a structured and reliable solution to complex data challenges.

Key Features of Apache Kafka:

Scalability: Kafka is designed to scale horizontally, allowing organizations to handle growing data volumes effortlessly. It can be distributed across multiple machines to ensure that even massive datasets are processed smoothly.
Fault Tolerance: Kafka is built to withstand system failures. It replicates data across multiple brokers, ensuring that if one broker fails, another can take over without data loss.
Low Latency: Kafka provides real-time data streaming with minimal delays, making it ideal for applications that require up-to-the-second data processing.
High Throughput: Kafka is optimized for handling high-throughput data streams. It can process millions of records per second, making it suitable for use cases that rapidly generate a large amount of data.

Why Use Apache Kafka?

Unlike traditional message brokers or ETL(Extracting-Transforming-Loading) tools, Kafka manages continuous, high-volume data streams with low latency while providing a fault-tolerant, scalable architecture.

Traditional systems often struggle with maintaining performance at scale or processing real-time data efficiently, whereas Kafka is specifically designed for these challenges.

Here's a comparison table highlighting the differences between Apache Kafka and traditional message brokers or ETL tools:

Feature	Apache Kafka	Traditional Message Brokers / ETL Tools
Data Handling	Real-time stream processing	Typically batch-oriented or slower messaging
Scalability	Horizontally scalable with distributed architecture	Often limited in scaling capabilities
Fault Tolerance	Built-in data replication and durability	Limited fault tolerance, often relies on external systems for backup
Latency	Low latency (real-time processing)	Higher latency, especially in batch processing
Throughput	High throughput, capable of handling millions of messages per second	May struggle with high-volume data at scale
Flexibility	Can process both stream and batch data	Often optimized for either batch or real-time, not both
Use Cases	Real-time analytics, IoT, event-driven architectures	Messaging, batch processing, and simpler integrations
Data Storage	Persistent storage of data streams	Generally does not store data for long durations

Having understood where Kafka can be helpful and should be used, let's look at its core components next.

Core Components of Apache Kafka

1. Topics:

A topic is a logical channel to which Kafka producers send data and from which consumers read data.

Each message in Kafka belongs to a specific topic, and topics act as the primary mechanism for organizing and categorizing data.

2. Partitions:

Each topic in Kafka can be divided into partitions, which are the basic units of parallelism and scalability.

Partitions allow Kafka to distribute the load across multiple brokers, enabling the system to handle large volumes of data.

Each partition is an ordered, immutable sequence of messages, and messages within a partition are assigned a unique offset.

3. Producers:

A producer is any application or service that sends data to Kafka topics.

Producers push data to topics, typically in the form of messages or events, and they can write to one or more partitions within those topics.

4. Consumers:

A consumer is any application or service that reads data from Kafka topics.

Consumers subscribe to one or more topics and can consume messages from one or more partitions. Multiple consumers can work in parallel to handle high throughput.

But imagine a system processing thousands of real-time orders every second. A single consumer would quickly become overwhelmed, leading to delays and failures. This is where we have consumer groups.

A consumer group is a group of consumers that work together to read data from a topic. Kafka automatically balances the workload among consumers in a group by assigning each partition to one consumer.

If a topic has multiple partitions, a consumer group can have multiple consumers reading from those partitions in parallel, which speeds up data processing.
If one consumer in the group fails, Kafka reassigns its partition to another consumer in the group, ensuring continuous processing.

5. Brokers:

A broker is a Kafka server that stores and serves data. Kafka clusters consist of multiple brokers, with each broker managing a subset of partitions. Brokers store messages and ensure data replication and fault tolerance.

6. Zookeeper:

Zookeeper is a distributed coordination service used in older versions of Kafka to manage cluster metadata, leader election, and configuration. It ensures consistency and synchronization across Kafka brokers.

However, with the introduction of KRaft (Kafka Raft) mode, Kafka is gradually moving towards eliminating the reliance on Zookeeper. KRaft allows Kafka to manage metadata directly, simplifying the architecture.

Message Flow in Kafka

1. Producers Send Messages to Topics

Producers are applications or systems that generate data and send it to Kafka. Producers write messages (data) to a specific topic in Kafka. A topic acts like a category or a label to organize messages.

Example: A ride-sharing app might send ride requests to a topic called ride-requests.

2. Partitioning and Replication

Partitioning: Each topic is split into smaller parts called partitions. Messages within a partition are stored in the order they arrive. Kafka assigns each message a unique ID called an offset, which helps keep track of messages.

Partitioning allows Kafka to handle large amounts of data by distributing it across multiple servers (brokers).

Example: If the topic ride-requests has 3 partitions, messages like Request A, Request B, and Request C may go to Partition 1, Partition 2, and Partition 3.

Replication: To ensure reliability, Kafka makes copies of partitions across multiple brokers. This means that even if one broker fails, another can serve the data, preventing data loss.

3. Consumers Read Messages Using Offsets

Consumers are applications that read messages from topics. Each consumer keeps track of the offset of the last message it read. This ensures that consumers process messages in the correct order and don’t miss any data.

Example: A consumer might read ride requests from the ride-requests topic and process them to match drivers with passengers.

Real-World Applications of Apache Kafka

Apache Kafka is a versatile platform used across industries to handle real-time data at scale. Here are some key applications:

Industry Use Cases

E-commerce Platforms:
- Order Tracking: Stream real-time order updates to customers and internal systems.
- Inventory Management: Monitor stock levels and synchronize inventory across warehouses instantly.
Finance:
- Fraud Detection: Process transactional data in real-time to identify unusual activity.
- Trade Monitoring: Track and analyze stock trades to meet regulatory requirements and optimize strategies.
Social Media Platforms:
- Activity Streams: Stream real-time user activity like likes, comments, and shares for personalized feeds.
- Notifications: Deliver instant notifications based on user actions or system alerts.
IoT Applications:
- Sensor Data Ingestion: Collect and process data from IoT devices like temperature sensors or fitness trackers.
- Device Communication: Enable real-time communication between connected devices, such as smart home systems.

Conclusion

Apache Kafka is a distributed platform for real-time data streaming that transforms how businesses handle data. Let’s recap its core aspects:

Definition: Kafka enables organizations to publish, subscribe to, store, and process streams of records in real-time.
Components: Key elements include topics, partitions, producers, consumers, brokers, and (optionally) Zookeeper.
Workflow: Producers send data to topics, partitions distribute the load, brokers replicate data for fault tolerance, and consumer groups process data in parallel using offsets for tracking.
Benefits: Kafka provides scalability, fault tolerance, low latency, and support for high-throughput workloads.

The next article will discuss how to set up Apache Kafka locally, create topics, partitions, producers and consumers, and send messages across them. Keep reading!

Setting Up Apache Kafka Locally: A Step-by-Step Guide

Pragati Verma ・ Jan 21

#computerscience #database #webdev #tutorial

Scraping Infinite Scroll Pages with a 'Load More' Button: A Step-by-Step Guide

Pragati Verma — Mon, 13 Jan 2025 08:29:48 +0000

Are your scrapers stuck when trying to load data from dynamic web pages? Are you frustrated with infinite scrolls or those pesky "Load more" buttons?

You're not alone. Many websites today implement these designs to improve user experience—but they can be challenging for web scrapers.

This tutorial will guide you through a beginner-friendly walkthrough for scraping a demo page with a Load More button. Here’s what the target web page looks like:

By the end, you'll learn how to:

Set up Selenium for web scraping.
Automate the "Load more" button interaction.
Extract product data such as names, prices, and links.

Let's dive in!

Step 1: Prerequisites

Before diving in, ensure the following prerequisites:

Python Installed: Download and install the latest Python version from python.org, including pip during setup.
Basic Knowledge: Familiarity with web scraping concepts, Python programming, and working with libraries such as requests, BeautifulSoup, and Selenium.

Libraries Required:

Requests: For sending HTTP requests.
BeautifulSoup: For parsing the HTML content.
Selenium: For simulating user interactions like button clicks in a browser.

You can install these libraries using the following command in your terminal:

pip install requests beautifulsoup4 selenium

Before using Selenium, you must install a web driver matching your browser. For this tutorial, we'll use Google Chrome and ChromeDriver. However, you can follow similar steps for other browsers like Firefox or Edge.

Install the Web Driver

Check your browser version:
Open Google Chrome and navigate to Help > About Google Chrome from the three-dot menu to find the Chrome version.
Download ChromeDriver:
Visit the ChromeDriver download page.
Download the driver version that matches your Chrome version.
Add ChromeDriver to your system PATH:
Extract the downloaded file and place it in a directory like /usr/local/bin (Mac/Linux) or C:\Windows\System32 (Windows).

Verify Installation

Initialize a Python file scraper.py in your project directory and test that everything is set up correctly by running the following code snippet:

from selenium import webdriver
driver = webdriver.Chrome() # Ensure ChromeDriver is installed and in PATH
driver.get("https://www.scrapingcourse.com/button-click")
print(driver.title)
driver.quit()

You can execute the above file code by running the following command on your terminal:

python scraper.py

If the above code runs without errors, it will spin up a browser interface and open the demo page URL as shown below:

Selenium will then extract the HTML and print the page title. You will see an output like this -

Load More Button Challenge to Learn Web Scraping - ScrapingCourse.com

This verifies that Selenium is ready to use. With all requirements installed and ready to use, you can start accessing the demo page's content.

Step 2: Get Access to the Content

The first step is to fetch the page's initial content, which gives you a baseline snapshot of the page's HTML. This will help you verify connectivity and ensure a valid starting point for the scraping process.

You will retrieve the HTML content of the page URL by sending a GET request using the Requests library in Python. Here's the code:

import requests
# URL of the demo page with products
url = "https://www.scrapingcourse.com/button-click"
# Send a GET request to the URL
response = requests.get(url)
# Check if the request was successful
if response.status_code == 200:
    html_content = response.text
    print(html_content) # Optional: Preview the HTML
else:
    print(f"Failed to retrieve content: {response.status_code}")

The above code will output the raw HTML containing the data for the first 12 products.

This quick preview of the HTML ensures that the request was successful and that you're working with valid data.

Step 3: Load More Products

To access the remaining products, you'll need to programmatically click the "Load more" button on the page until no more products are available. Since this interaction involves JavaScript, you will use Selenium to simulate the button click.

Before writing code, let’s inspect the page to locate:

The "Load more" button selector (load-more-btn).
The div holding the product details (product-item).

You'll get all the products by loading more products, giving you a larger dataset by running the following code:

from selenium import webdriver
from selenium.webdriver.common.by import By
import time
# Set up the WebDriver (make sure you have the appropriate driver installed, e.g., ChromeDriver)
driver = webdriver.Chrome()
# Open the page
driver.get("https://www.scrapingcourse.com/button-click")
# Loop to click the "Load More" button until there are no more products
while True:
    try:
        # Find the "Load more" button by its ID and click it
        load_more_button = driver.find_element(By.ID, "load-more-btn")
        load_more_button.click()
        # Wait for the content to load (adjust time as necessary)
        time.sleep(2)
    except Exception as e:
        # If no "Load More" button is found (end of products), break out of the loop
        print("No more products to load.")
        break
# Get the updated page content after all products are loaded
html_content = driver.page_source
# Close the browser window
driver.quit()

This code opens the browser, navigates to the page, and interacts with the "Load more" button. The updated HTML, now containing more product data, is then extracted.

If you don’t want Selenium to open the browser every time you run this code, it also provides headless browser capabilities. A headless browser has all the functionalities of an actual web browser but no Graphical User Interface (GUI).

You can enable the headless mode for Chrome in Selenium by defining a ChromeOptions object and passing it to the WebDriver Chrome constructor like this:

from selenium import webdriver
from selenium.webdriver.common.by import By

import time

# instantiate a Chrome options object
options = webdriver.ChromeOptions()

# set the options to use Chrome in headless mode
options.add_argument("--headless=new")

# initialize an instance of the Chrome driver (browser) in headless mode
driver = webdriver.Chrome(options=options)

...

When you run the above code, Selenium will launch a headless Chrome instance, so you’ll no longer see a Chrome window. This is ideal for production environments where you don’t want to waste resources on the GUI when running the scraping script on a server.

Now that the complete HTML content is retrieved extracting specific details about each product is time.

Step 4: Parse Product Information

In this step, you'll use BeautifulSoup to parse the HTML and identify product elements. Then, you'll extract key details for each product, such as the name, price, and links.

from bs4 import BeautifulSoup
# Parse the page content with BeautifulSoup
soup = BeautifulSoup(html_content, 'html.parser')
# Extract product details
products = []
# Find all product items in the grid
product_items = soup.find_all('div', class_='product-item')
for product in product_items:
    # Extract the product name
    name = product.find('span', class_='product-name').get_text(strip=True)

    # Extract the product price
    price = product.find('span', class_='product-price').get_text(strip=True)

    # Extract the product link
    link = product.find('a')['href']

    # Extract the image URL
    image_url = product.find('img')['src']

    # Create a dictionary with the product details
    products.append({
        'name': name,
        'price': price,
        'link': link,
        'image_url': image_url
})
# Print the extracted product details
for product in products[:2]:
    print(f"Name: {product['name']}")
    print(f"Price: {product['price']}")
    print(f"Link: {product['link']}")
    print(f"Image URL: {product['image_url']}")
    print('-' * 30)

In the output, you should see a structured list of product details, including the name, image URL, price, and product page link, like this -

Name: Chaz Kangeroo Hoodie
Price: $52
Link: https://scrapingcourse.com/ecommerce/product/chaz-kangeroo-hoodie
Image URL: https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh01-gray_main.jpg
------------------------------
Name: Teton Pullover Hoodie
Price: $70
Link: https://scrapingcourse.com/ecommerce/product/teton-pullover-hoodie
Image URL: https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh02-black_main.jpg
------------------------------
…

The above code will organize the raw HTML data into a structured format, making it easier to work with and preparing the output data for further processing.

Step 5: Export Product Information to CSV

You can now organize the extracted data into a CSV file, which makes it easier to analyze or share. Python's CSV module helps with this.

import csv
# Write the product information to a CSV file
with open("products.csv", mode="w", newline="") as file:
    writer = csv.DictWriter(file, fieldnames=["name", "image_url", "price", "link"])
    writer.writeheader()
    for product in products:
        writer.writerow(product)

The above code will create a new CSV file with all the required product details.

Here's the complete code for an overview:

from selenium import webdriver
from selenium.webdriver.common.by import By
from bs4 import BeautifulSoup
import time
import csv
# Set up the WebDriver (make sure you have the appropriate driver installed, e.g., ChromeDriver)
driver = webdriver.Chrome()
# Open the page
driver.get("https://www.scrapingcourse.com/button-click")
# Loop to click the "Load More" button until there are no more products to load
while True:
    try:
        # Find the "Load more" button by its ID and click it
        load_more_button = driver.find_element(By.ID, "load-more-btn")
        load_more_button.click()
        # Wait for the content to load (adjust time as necessary)
        time.sleep(2)
    except Exception as e:
        # If no "Load More" button is found (end of products), break out of the loop
        print("No more products to load.")
        break
# Get the updated page content after all products are loaded
html_content = driver.page_source
# Close the browser window
driver.quit()
# Parse the page content with BeautifulSoup
soup = BeautifulSoup(html_content, 'html.parser')
# Extract product details
products = []
# Find all product items in the grid
product_items = soup.find_all('div', class_='product-item')
for product in product_items:
    # Extract the product name
    name = product.find('span', class_='product-name').get_text(strip=True)

    # Extract the product price
    price = product.find('span', class_='product-price').get_text(strip=True)

    # Extract the product link
    link = product.find('a')['href']

    # Extract the image URL
    image_url = product.find('img')['src']

    # Create a dictionary with the product details
    products.append({
        'name': name,
        'price': price,
        'link': link,
        'image_url': image_url
    })
# Print the extracted product details
for product in products[:2]: # You can modify the slice as needed to check more products
    print(f"Name: {product['name']}")
    print(f"Price: {product['price']}")
    print(f"Link: {product['link']}")
    print(f"Image URL: {product['image_url']}")
    print('-' * 30)
# Write the product information to a CSV file
with open("products.csv", mode="w", newline="") as file:
    writer = csv.DictWriter(file, fieldnames=["name", "image_url", "price", "link"])
    writer.writeheader()
    for product in products:
        writer.writerow(product)

The above code will create a products.csv which would look like this:

name,image_url,price,link
Chaz Kangeroo Hoodie,https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh01-gray_main.jpg,$52,https://scrapingcourse.com/ecommerce/product/chaz-kangeroo-hoodie
Teton Pullover Hoodie,https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh02-black_main.jpg,$70,https://scrapingcourse.com/ecommerce/product/teton-pullover-hoodie
Bruno Compete Hoodie,https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh03-black_main.jpg,$63,https://scrapingcourse.com/ecommerce/product/bruno-compete-hoodie
…

Step 6: Get Extra Data for Top Products

Now, let's say you want to identify the top 5 highest-priced products and extract additional data (such as the product description and SKU code) from their individual pages. You can do that using the code as follows:

# Sort products by price in descending order
sorted_products = sorted(products, key=lambda x: float(x['price'].replace('$', '')), reverse=True)
# Scrape extra details for the top 5 products
driver = webdriver.Chrome()
for product in sorted_products[:5]:
    driver.get(product['link'])
    time.sleep(3)
    soup = BeautifulSoup(driver.page_source, 'html.parser')
    description = soup.find('div', class_='product-description')
    product['description'] = description.get_text(strip=True) if description else "No description"
    sku = soup.find('span', class_='sku')
    product['sku'] = sku.get_text(strip=True) if sku else "No SKU"
driver.quit()

Here's the complete code for an overview:

from selenium import webdriver
from selenium.webdriver.common.by import By
from bs4 import BeautifulSoup
import time
import csv
# Set up the WebDriver (make sure you have the appropriate driver installed, e.g., ChromeDriver)
driver = webdriver.Chrome()
# Open the page
driver.get("https://www.scrapingcourse.com/button-click")
# Loop to click the "Load More" button until there are no more products to load
while True:
    try:
        # Find the "Load more" button by its ID and click it
        load_more_button = driver.find_element(By.ID, "load-more-btn")
        load_more_button.click()
        # Wait for the content to load (adjust time as necessary)
        time.sleep(2)
    except Exception as e:
        # If no "Load More" button is found (end of products), break out of the loop
        print("No more products to load.")
        break
# Get the updated page content after all products are loaded
html_content = driver.page_source

# Parse the page content with BeautifulSoup
soup = BeautifulSoup(html_content, 'html.parser')
# Extract product details
products = []
# Find all product items in the grid
product_items = soup.find_all('div', class_='product-item')
for product in product_items:
    # Extract the product name
    name = product.find('span', class_='product-name').get_text(strip=True)

    # Extract the product price
    price = product.find('span', class_='product-price').get_text(strip=True)

    # Convert price to a float for sorting (remove '$' or other symbols as needed)
    price_float = float(price.replace('$', '').replace(',', '').strip())

    # Extract the product link
    link = product.find('a')['href']

    # Extract the image URL
    image_url = product.find('img')['src']

    # Create a dictionary with the product details
    products.append({
        'name': name,
        'price': price,
        'price_float': price_float, # store as float for sorting
        'link': link,
        'image_url': image_url
})
# Sort products by price (descending order)
sorted_products = sorted(products, key=lambda x: x['price_float'], reverse=True)
# Get the top 5 highest-priced products
top_5_products = sorted_products[:5]
# Visit each of the top 5 product pages and extract extra data
for product in top_5_products:
    # Open the product page
    driver.get(product['link'])
    time.sleep(3) # Wait for the page to load
    # Parse the page content of the product page
    product_page_soup = BeautifulSoup(driver.page_source, 'html.parser')
    # Extract product description and SKU
    description = product_page_soup.find('div', class_='product-description')
    description_text = description.get_text(strip=True) if description else 'No description available'
    sku = product_page_soup.find('span', class_='sku')
    sku_code = sku.get_text(strip=True) if sku else 'No SKU available'
    # Add the extra data to the product details
    product['description'] = description_text
    product['sku'] = sku_code
# Close the browser window after scraping product pages
driver.quit()
# Print the extracted product details for the top 5 products
for product in top_5_products:
    print(f"Name: {product['name']}")
    print(f"Price: {product['price']}")
    print(f"Link: {product['link']}")
    print(f"Image URL: {product['image_url']}")
    print(f"Description: {product['description']}")
    print(f"SKU: {product['sku']}")
    print('-' * 30)
# Write the product information to a CSV file, including the extra data
with open("products.csv", mode="w", newline="") as file:
    writer = csv.DictWriter(file, fieldnames=["name", "image_url", "price", "link", "description", "sku"])
    writer.writeheader()
    for product in products:
        writer.writerow({
            'name': product['name'],
            'image_url': product['image_url'],
            'price': product['price'],
            'link': product['link'],
            'description': product.get('description', ''),
            'sku': product.get('sku', '')
})

This code sorts the products by price in descending order. Then, for the top 5 highest-priced products, the script opens their product pages and extracts the product description and SKU using BeautifulSoup.

The output of the above code will be like this:

Name: Lando Gym Jacket
Price: $99
Link: https://scrapingcourse.com/ecommerce/product/lando-gym-jacket
Image URL: https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj08-gray_main.jpg
Description: No description available
SKU: MJ08
------------------------------
Name: Ingrid Running Jacket
Price: $84
Link: https://scrapingcourse.com/ecommerce/product/ingrid-running-jacket
Image URL: https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/wj04-white_main.jpg
Description: No description available
SKU: WJ04
------------------------------
Name: Zeppelin Yoga Pant
Price: $82
Link: https://scrapingcourse.com/ecommerce/product/zeppelin-yoga-pant
Image URL: https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mp08-green_main.jpg
Description: No description available
SKU: MP08
------------------------------
Name: Zeppelin Yoga Pant
Price: $82
Link: https://scrapingcourse.com/ecommerce/product/zeppelin-yoga-pant
Image URL: https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mp08-green_main.jpg
Description: No description available
SKU: MP08
------------------------------
Name: Juno Jacket
Price: $77
Link: https://scrapingcourse.com/ecommerce/product/juno-jacket
Image URL: https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/wj06-purple_main.jpg
Description: No description available
SKU: No SKU available
------------------------------

The above code will update the products.csv and it will now have information like this:

name,image_url,price,link,description,sku
Chaz Kangeroo Hoodie,https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh01-gray_main.jpg,$52,https://scrapingcourse.com/ecommerce/product/chaz-kangeroo-hoodie,,
Teton Pullover Hoodie,https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh02-black_main.jpg,$70,https://scrapingcourse.com/ecommerce/product/teton-pullover-hoodie,,
Bruno Compete Hoodie,https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh03-black_main.jpg,$63,https://scrapingcourse.com/ecommerce/product/bruno-compete-hoodie,,
…

Conclusion

Scraping pages with infinite scrolling or "Load more" buttons can seem challenging, but using tools like Requests, Selenium, and BeautifulSoup simplifies the process.

This tutorial showed how to retrieve and process product data from a demo page, saving it in a structured format for quick and easy access.

See all the code snippets here.

Understanding JavaScript Deobfuscation in Web Scraping

Pragati Verma — Tue, 17 Dec 2024 07:30:33 +0000

Web scraping often involves extracting data from websites, but it can be challenging in many cases. The data you need might not be directly accessible due to obfuscation techniques used by websites.

JavaScript obfuscation, in particular, is a method used to make the scraping process more difficult using Javascript.

In this article, we’ll discuss JavaScript deobfuscation, why it matters in web scraping, and how to handle it.

What is JavaScript Obfuscation in Web Scraping?

JavaScript obfuscation is a technique for making code harder to interpret by humans or automated tools. Websites often obfuscate JavaScript to protect their data or hinder web scraping attempts.

Common obfuscation techniques can involve:

Changing Variable and Function Names: Variable and function names are renamed to random strings such as a1, b2 and so on, making it hard to understand their purpose.
Encoding Data: Strings, URLs or sensitive data are encrypted or encoded using base64 or other encoding methods.
Control Flow Obfuscation: The order of code execution can be altered to make it harder to follow the logic and flow of the program.
Dead Code Insertion: Irrelevant or unreachable code is added to increase the complexity and size of the script.
Minification: Unnecessary whitespace and comments are removed to reduce readability and shrink the code size.
Function Wrapping and Indirection: Important functions are wrapped in multiple layers or executed through indirect calls to add complexity and hide their true purpose.

Here’s a simple example of JavaScript obfuscation:

Original JS Code:

let productPrice = 29.99;
let productName = "Wireless Mouse";

function displayProductInfo() {
    console.log("Product Name: " + productName);
    console.log("Price: $" + productPrice);
}

displayProductInfo();

Obfuscated JS Code:

var _0x1a2b3c = ["\x57\x69\x72\x65\x6C\x65\x73\x73\x20\x4D\x6F\x75\x73\x65", "\x4C\x65\x74\x20\x70\x72\x6F\x64\x75\x63\x74\x50\x72\x69\x63\x65", "\x24"];
let _0x4c3b1a = 29.99;
let _0x6d24f5 = "Wireless Mouse";

function _0x44a5bc() {
    console[_0x1a2b3c[1]](_0x1a2b3c[0] + _0x6d24f5);
    console[_0x1a2b3c[1]](_0x1a2b3c[2] + _0x4c3b1a);
}

_0x44a5bc();

The JavaScript code may also alter requests, load additional data, or manage how the DOM (Document Object Model) is populated, making it harder to trace.

This is where Javascript deobfuscation comes in!

Why Is JavaScript Deobfuscation Important?

Deobfuscation is essential because many websites rely on obfuscation techniques to prevent scraping. To bypass these defenses, web scrapers must reverse the obfuscation process. Without deobfuscation, scrapers might miss crucial information or struggle to interact with dynamic elements properly.

In the context of scraping, understanding JavaScript deobfuscation allows you to:

Extract hidden or dynamically loaded data: Deobfuscation helps reveal data stored in JavaScript variables, encoded strings, or dynamically generated HTML, such as product prices, stock levels, or user reviews.
Interact with obfuscated JavaScript code: By reversing obfuscation techniques, you can better understand and interact with complex JavaScript code to access content that is otherwise hidden or manipulated.
Bypass anti-scraping measures: Deobfuscation helps overcome JavaScript-based anti-scraping mechanisms like CAPTCHAs, rate limiting, or browser fingerprinting, allowing scrapers to mimic human behaviour and bypass these defenses.

How Does JavaScript Deobfuscation Work?

Javascript deobfuscation refers to the process of interpreting or reversing obfuscated JavaScript code to reveal its original functionality.

Here are some of the common approaches for deobfuscating JavaScript:

Manual Inspection: Reviewing JavaScript code to identify patterns and decode obfuscated elements can be effective but is often time-intensive and demands a strong understanding of JavaScript.
Automated Deobfuscators: Tools like JSDetox or de4js can help you automate the process by detecting the common obfuscation patterns and helping you reverse them into more readable code.
Debugging Tools: Web scraping developers can use browser developer tools to step through JavaScript code and observe its execution. This helps trace how the script manipulates the page or communicates with the server.
Code Beautifiers: These tools format obfuscated code to make it more readable, which is often the first step before more complex deobfuscation techniques are applied.

Solutions for Dealing with Obfuscated JavaScript

Headless Browsers: Tools like Puppeteer or Playwright help deal with JavaScript-heavy websites. These headless browsers can execute JavaScript and allow scrapers to interact with the page as if they’re a real user, helping to bypass complex obfuscation techniques.
Cloudflare Workarounds: Some websites use Cloudflare to protect against scraping. Scrapers can bypass Cloudflare's anti-bot protection by rotating user-agent headers or solving JavaScript challenges. The most common is the Cloudflare JS Challenge.
Dynamic Data Scraping: Many scraping tools are designed to replicate user behaviour on JavaScript-heavy websites, allowing them to interact with elements like buttons or drop-downs to load dynamic content.

Conclusion

JavaScript deobfuscation is crucial in web scraping, enabling you to bypass obfuscation techniques and access the data needed. While manual deobfuscation can be time-consuming, automated tools and headless browsers make the process more efficient.

The Complete Guide to Logging in Django

Pragati Verma — Thu, 18 Apr 2024 13:51:44 +0000

Applications running in production can’t tell you directly what’s going on under the hood, so you need a way to keep track of their behavior. Knowing what’s going on in your code at any given time benefits you from both a technical and a business standpoint. With this information, engineers and product managers can make better decisions about which systems to repair or how to improve user experience (UX).

The most straightforward way to achieve this is with logging. You can code the program during development to share relevant information while running that could be useful in analysis, debugging, and further troubleshooting.

Django is a popular Python web application framework, used by small and large organizations. Python offers a powerful built-in logging module to log events from applications and libraries. The library is flexible and can be easily customized to fit any application. You can implement it this way:

import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

logger.info('Start reading database')
# read database here
records = {'john': 55, 'tom': 66}
logger.debug('Records: %s', records)
logger.info('Updating records ...')
# update records here
logger.info('Finish updating records')

When the program is run, you can see the records on the console:

INFO:__main__:Start reading database
INFO:__main__:Updating records ...
INFO:__main__:Finish updating records

This article will delve deeper into the logging concepts that every developer should know. You’ll learn how to log significant data, how to route the logs, and how to consolidate them for insights into your Django applications. You’ll also learn best practices to follow.

What Are Logs?

Logs are the records of events that occur while running software. They contain information about the application, system performance, or user activities. Loggers are the objects that a developer interacts with to print out the information. They help you tell the program what to log and how to do it.

Adding logging to your code involves three critical steps:

Choosing what data to output and where in the code to do so
Choosing how to format the logs
Choosing where to transmit the logs (e.g., stdout or syslog)

How to Implement Logging

To implement logging in a Django application, you must consider the following factors.

Logging Levels

The Python logging library adds several types of metadata to provide valuable context around the log messages. This can help in diagnosing a problem or analyzing what’s happening inside the code while the application is running. For example, log levels define the severity level of log events, which can be used to segment logs so you get the most relevant log message at any specified time.

You can use log levels to help prioritize log messages. For instance, when you’re developing an application, DEBUG information is most relevant; while you’re running the application, you can leave INFO logs to indicate certain events.

The Python logging package comes with five logging levels: critical, error, warning, info, and debug. These levels are denoted by constants with the same name: logging.CRITICAL, logging.ERROR, logging.WARNING, logging.INFO, and logging.DEBUG, with values of 50, 40, 30, 20, and 10. A level’s meaning is determined at runtime by its value.

The community-wide applicability rules for logging levels are as follows:

DEBUG: logging.DEBUG can be used to log detailed information for debugging code in development, such as when the app starts.
INFO: logging.INFO can be used to log information about the code if it is running as expected, such as when a process starts in the app.
WARNING: logging.WARNING can be used to report unexpected behavior that could cause a future problem but isn’t impacting the current process of the application, such as when the app detects low memory.
ERROR: logging.ERROR can be used to report events when the software fails to perform some action, such as when the app fails to save data due to insufficient permissions given to the user.
CRITICAL: logging.CRITICAL can be used to report serious errors that impact the continued execution of the application, such as when the application fails to store data due to insufficient memory.

The logging.Logger object offers the primary interface to Python’s logging library. These objects include methods for issuing log requests and for querying and modifying their state, as follows:

Logger.critical(msg, *args, **kwargs)
Logger.error(msg, *args, **kwargs)
Logger.debug(msg, *args, **kwargs)
Logger.info(msg, *args, **kwargs)
Logger.warn(msg, *args, **kwargs)

In addition, loggers provide the following two options:

Logger.log(level, msg, *args, **kwargs) sends log requests with defined logging levels. When you’re using custom logging levels, this approach comes in handy.
Logger.exception(msg, *args, **kwargs) sends log requests with the logging level ERROR and includes the current exception in the log entries. This function should be called from an exception handler.

Logging Handlers

Logging handlers are used to determine where to put the logs—system logs or files. Unless explicitly specified, the logging library uses a StreamHandler for sending the log messages to the console or sys.stderr.

Handlers also format log records into log entries using their formatters. The formatter for a handler can be set by clients using the Handler.setFormatter(formatter) function. If a handler doesn’t have a formatter, the library’s default formatter is used.

The logging.handler module includes fifteen useful handlers that span a range of use cases (including the ones mentioned above). The most commonly used logging handlers are as follows:

StreamHandler transmits logs to a stream-like object, such as a console, using stdout.
FileHandler redirects log events to a file.
SyslogHandler routes logs to your system’s syslog daemon.
HTTPHandler allows you to deliver logs through HTTP.
NullHandler redirects your logs to nowhere, which is helpful for temporarily halting logging.

Logging Formatters

Log messages from the logging library follow this default format:

<LEVEL>:<LOGGER_NAME>:<MESSAGE>

However, they can be customized to add more information using logging.Formatter objects to change them into a string-based log entry.

Django leverages the potential of logging by using the Python logging module by default, which provides different ways to create customized loggers through handlers, formats, and levels. The logging module is capable of:

Multithreading execution
Categorizing messages via different log levels
Setting the destination of the logs
Controlling what to include and what to emit

How to Add Logging in Django

To use logging in your Django project, follow these steps.

Configure the settings.py for various loggers, handlers, and filters:

import logging.config
import os
from django.utils.log import DEFAULT_LOGGING

# Disable Django's logging setup
LOGGING_CONFIG = None

LOGLEVEL = os.environ.get('LOGLEVEL', 'info').upper()

logging.config.dictConfig({
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'default': {
            # exact format is not important, this is the minimum information
            'format': '%(asctime)s %(name)-12s %(levelname)-8s %(message)s',
        },
        'django.server': DEFAULT_LOGGING['formatters']['django.server'],
    },
    'handlers': {
        # console logs to stderr
        'console': {
            'class': 'logging.StreamHandler',
            'formatter': 'default',
        },
        'django.server': DEFAULT_LOGGING['handlers']['django.server'],
    },
    'loggers': {
        # default for all undefined Python modules
        '': {
            'level': 'WARNING',
            'handlers': ['console'],
        },
        # Our application code
        'app': {
            'level': LOGLEVEL,
            'handlers': ['console'],
            # Avoid double logging because of root logger
            'propagate': False,
        },
        # Default runserver request logging
        'django.server': DEFAULT_LOGGING['loggers']['django.server'],
    },
})

Restart the server. You’ll be able to see logs on your console or log files, depending on the configuration.

Best Practices for Logging in Django

Logging is vital to your Django application because it can save you time in crucial situations. Follow these best practices for implementation:

Create loggers using the logging.getLogger() factory function, so that the logging library can manage the mapping of names to instances and maintain a hierarchy of logs. This way, you can use the logger’s name to access it in different parts of the application, and only a set number of loggers will be created at runtime.
Specify proper log levels to lower the risks of deployment and ensure effective debugging. This helps prevent the flooding of log files with trivial information due to inappropriate settings.
Format logs correctly so that the system can parse them. This is useful when manually reading the logs isn’t enough, such as for audits or alerts.
Don’t log sensitive information like passwords, authorization tokens, personally identifiable information (PII), credit card numbers, or session identifiers that the user has chosen.
Use fault-tolerant protocols while transferring logs to avoid packet loss. Secure log data by encrypting it and removing any sensitive information before transferring it.
Create meaningful log messages so that you can easily tell what happened from the log file.
Enhance log messages by including additional information.
Don’t make log messages reliant on the content of prior messages, since the previous messages may not display if they’re logged at different levels.
Ensure that logs are written asynchronously during log generation. Buffer or queue logs to prevent the program from stalling. Organize logs so that it’s simple to make changes as needed.
Use a wrapper to shield the program from third-party tools. Use standard date and time formats, add timestamps, use log levels appropriately, and include a stack trace when reporting the error to make logs more human-readable. Include the thread’s name in a multithreaded program.
Use filters or logging.LoggerAdapter to inject local contextual information, or use logging.setLogRecordFactory() to inject global contextual information in the log records. Don’t log too much, though, or it might become difficult to extract value.
If you use FileHandler to write logs, the log file will expand over time and eventually take up all your storage space. In the production environment, utilize RotatingFileHandler instead of FileHandler to prevent this problem.
When you have many different servers and log files, set up a central log system for all critical messages so you can quickly monitor it for problems.

Logging configuration in Django is simple, but it can get complicated when dealing with large applications. Aside from the Python logging module, you can also use popular logging tools such as NewRelic.

Final Thoughts

Logging can help you improve your application development and end-user experience. Because Django is a Python-based framework, Python’s logging system benefits you in several ways, and you can implement it fairly easily. Remember to follow the above best practices to simplify your setup and ensure better-quality results.

How to Handle N+1 Queries for Optimal Database Performance in Django?

Pragati Verma — Mon, 01 Apr 2024 14:34:33 +0000

As a backend developer, prioritizing efficiency is the foremost metric for your application's success. Every database query has implications for its performance, scalability, and overall responsiveness, and one of the most common issues developers face is the N+1 query problem.

N+1 queries can lead to sluggish user experience, decreased system throughput, and even scalability challenges as the volume of the data and user interactions increase over time, thus, addressing N+1 queries is very important for ensuring optimal database performance and responsiveness.

In this article, we'll learn what N+1 queries are, how to find them, and finally, how to resolve them in Django. So, let's dive right in!

What are N+1 Queries?

N+1 query refers to a common scenario in database management where an initial query to retrieve data is followed by N additional queries to fetch related data for each result of the initial query, resulting in a cascade of queries that leads to potentially significant performance overhead.

Imagine you're at a grocery store and you need to buy ingredients for a recipe. The N+1 problem is like going to the store multiple times, once for each ingredient, instead of making a single trip and getting everything you need at once. It's inefficient and takes more time and effort than necessary. Similarly, in programming, N+1 queries involve making multiple database queries for related data instead of optimizing and fetching everything in one go, leading to slower performance and wasted resources.

This problem usually arises when using an ORM(Object-Relational Mapping) in web frameworks such as Django or Ruby on Rails, and can impact the performance and scalability of the applications.

The most common occurrence of N+1 queries is when nested loops are used to iterate over the query results and fetch the related data within the inner loops. As you can imagine, this can lead to multiple queries being executed for each iteration of the outer loop.

Let's see an example. Suppose we have two models: Category and Product, where each product belongs to a category.

# models.py
from django.db import models

class Category(models.Model):
    name = models.CharField(max_length=100)

class Product(models.Model):
    name = models.CharField(max_length=100)
    category = models.ForeignKey(Category, related_name='products', on_delete=models.CASCADE)
    price = models.DecimalField(max_digits=10, decimal_places=2)

Now, let's say we want to retrieve all products along with their categories' names:

# views.py
from .models import Product

def get_product_list():
    products = Product.objects.all()  # Initial query to fetch all products
    for product in products:
        print("Product:", product.name)
        print("Category:", product.category.name)  # Each access to category.name triggers an additional query

In the above code example, we retrieve all products and then iterate over each product to print its name and the name of its category.

However, accessing product.category.name for each product triggers an additional query to fetch the category's name, leading to the N+1 query problem.

Now, that we understand what N+1 queries are, let's see how to find them easily.

How to find N+1 Queries in Django?

If you're unaware of what to search for, the N+1 problem might slip under the radar during development and testing. It might only become evident in production when dealing with a larger database.

However, once identified, fixing N+1 queries is straightforward and can lead to significant performance enhancements for your application.

There are several ways to detect N+1 queries in your Django application -

Using APM tools like NewRelic, Sentry, Datadog, etc to monitor the performance of your application and while you're on it, they can help you identify N+1 queries.
Use the Django Debug Toolbar, a third-party debugging tool that provides detailed information about each HTTP request, including executed queries. Install and configure the debug toolbar in your Django project, and then inspect the SQL queries generated for each request. Look for patterns where multiple similar queries are executed, indicating potential N+1 query issues.
Enable SQL query logging in your Django settings. Set the django.db.backends logger to the DEBUG level to log all executed SQL queries. Inspect the logged queries, and look for patterns where multiple similar queries are executed in succession, indicating potential N+1 query issues.

Now, that we know how to find N+1 queries, the next step is to learn how to fix them.

How to Fix N+1 Queries Once They're Identified?

Here are the ways to resolve N+1 queries in your Django codebase -

1. Use `select_related()` and `prefetch_related()`

An easy way to optimize queries and prevent N+1 queries is by employing joins. Joins enable the consolidation of related records into a single query, eliminating the need to query the database for each record separately. Using the joins method, you can specify the associations to be merged.

In Django, wherever applicable, you can use Django's QuerySet methods select_related() and prefetch_related() to optimize data retrieval.

select_related(): Use when fetching related objects via foreign key relationships. It performs an SQL join operation to fetch related objects in a single query.

For example, to fix the N+1 query when fetching the product list, we can select_related like this -

# views.py
from .models import Product

def get_product_list():
    products = Product.objects.select_related('category').all()
    for product in products:
        print("Product:", product.name)
        print("Category:", product.category.name)  # Fixed

prefetch_related(): Use when fetching related objects via many-to-many or reverse foreign key relationships. It fetches related objects in separate queries and caches them for efficient retrieval.

Now, suppose we have a view that fetches all categories along with their related products:

# views.py
from .models import Category

def get_category_list():
    categories = Category.objects.all()
    for category in categories:
        print("Category:", category.name)
        for product in category.products.all():
            print("Product:", product.name)  # N+1 query issue

As you can see, we again have a scenario where N+1 queries would occur. To fix this, we can use prefetch_related to fetch the related Product objects in separate queries and cache them:

# views.py
from .models import Category

def get_category_list():
    categories = Category.objects.prefetch_related('products').all()
    for category in categories:
        print("Category:", category.name)
        for product in category.products.all():
            print("Product:", product.name)  # Fixed

2. Use `annotate()` to perform calculations

annotate() can be used to perform calculations across related models without requiring additional database queries for each related object.

It can help in solving N+1 queries by allowing you to aggregate or annotate related data into the initial queryset, thus reducing the need for subsequent queries to fetch related information.

Suppose we want to retrieve all categories along with the count of products in each category.

Without annotate(), if we retrieve all categories and then iterate over each category to count the number of products it contains, it would result in N+1 queries, where N is the number of categories.

# views.py
from .models import Category, Product

def get_category_list():
    categories = Category.objects.all()
    for category in categories:
        product_count = Product.objects.filter(category=category).count()  # N+1 queries issue
        print("Category:", category.name)
        print("Product Count:", product_count)

To solve the N+1 queries issue using annotate(), we can annotate the queryset with the count of products for each category:

# views.py
from django.db.models import Count
from .models import Category, Product

def category_list():
    categories = Category.objects.annotate(product_count=Count('product'))
    for category in categories:
        print("Category:", category.name)
        print("Product Count:", category.product_count)  # Fixed, no additional queries

3. Use `Subquery` for complex calculations

Subqueries can help in solving N+1 queries by allowing you to perform a separate query to fetch related data for each record in the initial queryset, without resorting to multiple queries in a loop. This can be particularly useful when the related data needs to be aggregated or filtered based on each record in the initial queryset.

Let's use the Product and Category models again to demonstrate how subqueries can help in solving N+1 queries.

Suppose we want to retrieve all categories along with the total price of all products in each category.

Without using subqueries, if we retrieve all categories and then iterate over each category to calculate the total price of products, it would result in N+1 queries, where N is the number of categories.

# views.py
from .models import Category, Product

def get_category_list():
    categories = Category.objects.all()
    for category in categories:
        total_price = sum(product.price for product in category.products.all())  # N+1 queries issue
        print("Category:", category.name)
        print("Total Price:", total_price)

To solve the N+1 queries issue using subqueries, we can use a subquery to annotate the queryset with the total price of products for each category.

# views.py
from django.db.models import Subquery, Sum
from .models import Category, Product

def category_list():
    categories = Category.objects.annotate(
        total_price=Subquery(
            Product.objects.filter(category=OuterRef('id')).values('category').annotate(total_price=Sum('price')).values('total_price'),
            output_field=models.DecimalField()
        )
    )
    for category in categories:
        print("Category:", category.name)
        print("Total Price:", category.total_price)  # Fixed, no additional queries

In the above example, we use a subquery to fetch the total price of products for each category. The Subquery() function performs a separate query for each category in the initial queryset, fetching the total price of products filtered by the category's primary key.

This allows us to annotate the Category queryset with the total price of products for each category in a single query, effectively solving the N+1 queries issue.

Conclusion

N+1 queries can really slow down your Django app. But don't worry! You can fix this issue by using simple tricks like select_related, prefetch_related, and subqueries. These techniques help you fetch related stuff efficiently, avoiding the N+1 query problem.

The interesting thing to understand at this point is that the most difficult part of the process is to find the N+1 queries while fixing them is quite simple.

Just keep an eye on your database queries and use the right methods to make your Django app run faster.

I hope you found this article useful. Do follow me for more insightful tech articles. In case you want to connect with me, follow the links below:

LinkedIn | GitHub | Twitter

Git Reflog: Rescuing Lost Work in Git's Timeline

Pragati Verma — Sat, 30 Mar 2024 14:34:25 +0000

Have you ever found yourself in a panic when you accidentally delete a commit or lose track of changes in your Git repository? Don't worry, you're not alone. Git, being the most powerful version control system, has a safety net called "Git Reflog" that can save the day.

In this article, we'll take a beginner-friendly dive into the world of Git Reflog and learn how it works, why it's essential, and how it can rescue your lost work in no time.

What is Git Reflog?

Git has a feature called "reference log," or simply "reflog," that maintains a chronological record of reference updates in a repository. These references include changes to branches, commits, and other pointers, such as the HEAD pointer which indicates the currently checked-out branch or commit. Each entry in the reflog represents a snapshot of the repository's state at a particular point in time, along with information about the action that caused the change.

Each log entry in a reflog file typically contains the following information:

Commit Hash: The hash of the commit being referenced or affected by the operation.
Previous Commit Hash: The hash of the commit before the operation was performed.
Action: The action that triggered the reference update (e.g., commit, checkout, reset).
Date and Time: The timestamp of when the operation occurred.
Author: The name and email address of the user who operated.

By default, reflogs retain a record of each HEAD position for the past 90 days.

It's important to note that reflog history is confined to the local repository and isn't accessible remotely.

These reflogs are stored in specific directories within the local repository's .git directory. These directories can be located at .git/logs/refs/heads/, .git/logs/HEAD, and also .git/logs/refs/stash if the Git stash feature has been utilized within the repository as Git maintains a separate reflog for its stash functionality.

Why Git Reflog is Needed?

The primary function of Git reflog is to act as a safety net, enabling developers to recover lost commits or branches, reverse accidental changes, and navigate their repository's history effortlessly.

In contrast to commands like git reset or git revert, which can irreversibly alter the repository's history or have broader impacts, Git reflog offers precise control. Developers can pinpoint specific moments in the repository's timeline and restore lost commits or branches without affecting unrelated parts of the history.

Additionally, Git reflog provides transparency by revealing the sequence of operations and changes made to references in the repository. This visibility is crucial for understanding the repository's evolution over time and diagnosing development issues effectively.

How to Use Git Reflog?

To use git reflog, you simply have to use either git reflog or git reflog show and you'll get an output like this -

a861fgd4b HEAD@{0}: commit: Initial Commit
06f00264a HEAD@{1}: commit: Add models
41d87a714 HEAD@{2}: commit: Add migrations
f80a505fb HEAD@{3}: commit: Add serializers
8dbf0c9f2 (origin/staging, staging) HEAD@{4}: pull: Fast-forward
62170e3c3 HEAD@{5}: pull: Fast-forward

Each reflog entry, in addition to its ordered index, is timestamped, facilitating filtering based on time.

To view the date and time information along with reflog entries in a more readable format, you can use the following option.

git reflog --date=local

This option displays the reflog entries with their corresponding timestamps in a local time format.

The output in this case would be like this -

72f3cbb50 HEAD@{Fri Mar 29 10:55:36 2024}: commit: Add missing created_at, updated_at
466c878c6 HEAD@{Thu Mar 28 16:15:21 2024}: commit: Update BaseModelAdmin
3cb9c945d HEAD@{Thu Mar 28 16:05:56 2024}: checkout: moving from ch/start-user-project to ch/minor-admin-changes

To see reflog entries with timestamps relative to the current time, rather than showing absolute timestamps with specific dates and times, you can use either of the following commands -

# To see activity on HEAD, including timestamp
git reflog --relative-date

#or

git reflog --date=relative

Relative timestamps make it easier to quickly understand when a particular action occurred relative to the current time.

The output, in this case, would look like this -

937136cbb HEAD@{3 weeks ago}: commit: Update trigger time
ffe27973b HEAD@{3 weeks ago}: commit: Update task name
c7b76e8f3 HEAD@{3 weeks ago}: commit: Add slack alert

You can also view the reflog entries for a specific branch(staging in my case) with relative timestamps. Here's how you can use it:

git reflog show --date=relative <branch_name>

The output of the above command would look something like this -

8dbf0c9f2 (origin/staging, staging) staging@{30 hours ago}: pull: Fast-forward
62170e3c3 staging@{31 hours ago}: pull: Fast-forward
34502ef1d staging@{3 days ago}: pull: Fast-forward
63c9ac44a (ch/ulj-weekly-application-summary-slack-message) staging@{4 days ago}: pull: Fast-forward
2c6bbf64f staging@{4 days ago}: pull: Fast-forward

By default, Git reflog provides insights into the history of the HEAD ref, which typically denotes the currently active branch in a repository.

However, Git also maintains reflogs for other references beyond just the HEAD. These references encompass branches, tags, remotes, and even the Git stash, offering a comprehensive record of recent actions across various parts of the repository.

The syntax for accessing a specific Git ref follows the pattern name@{qualifier}, allowing developers to pinpoint specific reference updates. For instance, otherbranch@{0} refers to the most recent update on the otherbranch.

The timestamp can also be used as a qualifier token, enabling precise retrieval of reflog entries based on some time-related criteria.

Commonly used time qualifiers include @{0} for the most recent update, @{6.minutes.ago} for actions six minutes prior, and @{3.day.ago} for events three days in the past. These qualifiers can be combined (e.g., 1.day.3.hours.ago) and accept plural forms (e.g., 5.minutes.ago).

For example, to view reflog entries for the develop branch from three days ago, you would execute -

git reflog develop@{3.days.ago}

#or

git reflog show develop@{3.days.ago}

This command retrieves reflog entries specifically related to the develop branch that occurred within the specified time window, enabling precise historical analysis and troubleshooting.

Common subcommands to git reflog

git reflog expire

The expire sub-command in Git helps clean up old or unreachable reflog entries, which can make your repository history tidier.

However, using this command without caution could accidentally delete important data. Usually, it's something Git handles internally rather than users directly dealing with it.

To play it safe, you can run a "dry run" with the -n or --dry-run option before actually deleting anything. This lets you see which entries would be deleted without actually removing them, acting like a safety net.

You can also set how long entries should stick around before expiring using the --expire=time option or by changing a setting called gc.reflogExpire.

git reflog delete

The delete command allows you to manually remove particular reflog entries. You can indicate which entry you want to delete by specifying its reference and qualifier.

However, just like the expire command, using delete can lead to unintended data loss and isn't commonly used by regular users.

What is the difference between git reflog and git log?

git reflog and git log are two components provided by Git that allow us to peek into the repository’s commit history, logs, and reflogs. Sometimes, they show similar histories, especially when a developer does a bunch of local commits without syncing with the remote server. This similarity can cause confusion between git reflog and git log.

However, they are different and have different uses.

The main difference is that the git log shows the public record of the repository's commit history, while git reflog is a private record specific to your workspace, showing your local commits.

When you push, fetch, or pull changes, the git log gets copied to the Git repository. But git reflog doesn't get copied, so you can't see it in the copied repository unless you have access to the computer where the local repository is stored.

The reflog is a file located in .git\logs\refs\heads that keeps track of local commits for a specific branch. It doesn't include commits that might have been deleted by Git garbage collection.

On the other hand, the git log shows a historical list of commits on a branch, starting from the most recent one and going back to the very first commit in the branch's history.

To summarize the differences between these two git commands, take a look at the following table -

Feature	git reflog	git log
Tracking	Tracks all reference updates (e.g., commits, branch operations, stash changes) in the local repository	Tracks the commit history of a specific branch or set of commits
Purpose	Provides a detailed history of recent actions, acting as a safety net for recovery	Presents a chronological list of commits with details such as messages, authors, dates, and hashes
Scope	Specific to the local repository	Reflects the commit history of the current branch or specified commits
Usage	Commonly used for troubleshooting, recovering lost work, and navigating recent changes	Typically used for reviewing project history, understanding changes, and tracking codebase evolution

Conclusion

In summary, Git maintains a reflog, which acts as a log of recent changes to your HEAD and branch references within the last few months (usually 90 days). This temporary history is continuously updated by Git whenever there's a modification to your branch tip.

Additionally, the reflog command offers functionality to manage this log. You can use the "expire" sub-command to remove outdated reflog entries, ensuring the log doesn't become cluttered with old data. Similarly, the "delete" sub-command allows you to specify and delete specific entries from the reflog when needed.

State Machines in Practice: Implementing Solutions for Real Challenges

Pragati Verma — Mon, 18 Mar 2024 17:55:57 +0000

If you've studied engineering, you've probably heard about State Machines. But beyond the theory, you might not have seen how they're used. Did you know that State Machines play a role in many everyday applications? In this article, we'll refresh our understanding of State Machines and explore their practical applications. Plus, we'll learn how to implement them quickly using Python. Ready to dive in? Let's go!

'State' is a common programming term that is experienced by all developers as they advance from beginning to intermediate-level programming. So, what exactly does the term "State" mean?

In general, an object's state is merely the current snapshot of the object or a portion of it. Meanwhile, in computer science, a program's state is defined as its position about previously stored inputs. In this context, the term "state" is used in the same manner as it is in science: the state of an object, such as a gas, liquid, or solid, represents its current physical nature, and the state of a computer program reflects its current values or contents.

The stored inputs are preserved as variables or constants in a computer program. While assessing the state of a program, developers might examine the values contained in these inputs. The state of the program may change while it runs - variables may change, and memory values may change. A control variable, such as one used in a loop, for example, changes the state of the program at each iteration. Examining the present state of a program may be used to test or analyze the codebase.

In simpler systems, state management is frequently handled with if-else, if-then-else, try-catch statements, or boolean flags; however, this is useless when there are too many states imaginable in a program. They can lead to clunky, complicated code that is difficult to understand, maintain, and debug.

One disadvantage of if-else-clauses or booleans is that they may get fairly extensive, and adding another state is difficult because it necessitates rewriting the code of many different classes.

Let's build a video player for example:

class Video:

   def __init__(self, source):
       self.source = source
       self.is_playing = False
       self.is_paused = False
       self.is_stopped = True

   # A video can only be played when paused or stopped
   def play(self):
       if not self.is_playing or self.is_paused:
           # Make the call to play the video
           self.is_playing = True
           self.is_paused = False
       else:
           raise Exception(
               'Cannot play a video that is already playing.'
           )

   # A video can only be paused when it is playing
   def pause(self):
       if self.is_playing:
           # Make the call to pause the video
           self.is_playing = False
           self.is_paused = True
       else:
           raise Exception(
               'Cannot pause a video that is not playing'
           )

   # A video can only be stopped when it is playing or paused
   def stop(self):
       if self.is_playing or self.is_paused:
           # Make the call to stop the video
           self.is_playing = False
           self.is_paused = False
       else:
           raise Exception(
               'Cannot stop a video that is not playing or paused'
           )

The above code snippet is an if-else implementation of a simple video player application, where the three basic states are - playing, paused, and stopped. However, if we try to add more states, the code will rapidly become complex, bloated, repetitive, and hard to understand and test. Let’s see what the code looks like when adding another state ‘rewind’:

class Video:
   def __init__(self, source):
       self.source = source
       self.is_playing = False
       self.is_paused = False
       self.is_rewinding = False
       self.is_stopped = True

   # A video can only be played when it is paused or stopped or rewinding
   def play(self):
       if self.is_paused or self.is_stopped or self.is_rewinding:
           # Make the call to play the video
           self.is_playing = True
           self.is_paused = False
           self.is_stopped = False
           self.is_rewinding = False
       else:
           raise Exception(
               'Cannot play a video that is already playing.'
           )

   # A video can only be paused when it is playing or rewinding
   def pause(self):
       if self.is_playing or self.is_rewinding:
           # Make the call to pause the video
           self.is_playing = False
           self.is_paused = True
           self.is_rewinding = False
           self.is_stopped = False
       else:
           raise Exception(
               'Cannot pause a video that is not playing or rewinding'
           )

   # A video can only be stopped when it is playing or paused or rewinding
   def stop(self):
       if self.is_playing or self.is_paused or self.is_rewinding:
           # Make the call to stop the video
           self.is_playing = False
           self.is_paused = False
           self.is_stopped = True
           self.is_rewinding = False
       else:
           raise Exception(
               'Cannot stop a video that is not playing or paused or rewinding'
           )

   # 4. A video can only be rewinded when it is playing or paused.
   def rewind(self):
       if self.is_playing or self.is_paused:
           # Make the call to rewind the video
           self.is_playing = False
           self.is_paused = False
           self.is_stopped = False
           self.is_rewinding = True
       else:
           raise Exception(
               'Cannot rewind a video that is not playing or paused'
           )

Without the state pattern, you'd have to examine the program's current state throughout the code, including the update and draw methods. If you want to add a fourth state, such as a settings screen, you'll have to update the code of many distinct classes, which is inconvenient. This is where the idea of state machines comes in handy.

What is a state machine?

State machines are not a novel concept in computer science; they are one of the basic design patterns utilized in the software business. It is more system-oriented than coding-oriented and is used to model around use cases.

Let's look at a simple real-life example of hiring a cab through Uber:

When you initially launch the program, it takes you to the home screen, where you type in your destination in the search area.
Once the right location has been identified, Uber displays recommended travel options such as Pool, Premier, UberGo, Uber XL, and others, along with a pricing estimate.
The trip is confirmed and a driver is assigned once you select the payment option and press the 'Confirm' button with the specified journey time if necessary.
Uber now displays a map on which you can locate your driver.

Screen 1 is the first screen that all users in this use case see, and it is self-contained. Screen 2 is reliant on Screen 1, and you will not be able to go to Screen 2 until you give accurate data on Screen 1. Likewise, screen 3 is dependent on screen 2, while screen 4 is dependent on screen 3. If neither you nor your driver cancels your trip, you'll be moved to screen 4, where you won't be able to plan another trip until your current one ends.

Let's say it's raining severely and no driver accepts your trip or no available driver in your region is found to finish your travel; an error notification warning you of driver unavailability shows, and you remain on screen 3. You may still return to screen 2, screen 1, and even the very first screen.

You are in a different step of the cab reservation process, and you may only go to the next level if a specified action in the current stage is successful. For example, if you input the wrong location on screen 1, you won't be able to proceed to screen 2, and you won't be able to proceed to screen 3 unless you pick a travel option on screen 2, but you may always return to the previous stage unless your trip is already booked.

In the above example, we've divided the cab booking process into several activities, each of which may or may not be authorized to call another activity based on the status of the booking. A state machine is used to model this. In principle, each of these stages/states should be autonomous, with one summoning the next only after the current one has been finished, successfully or otherwise.

In more technical words, the state machine enables us to split down a large complicated action into a succession of separate smaller activities, such as the cab booking activity in the preceding example.

Events connect smaller tasks, and shifting from one state to another is referred to as transition. We normally conduct some actions after changing from one state to another, such as creating a booking in the back end, issuing an invoice, saving user analytics data, capturing booking data in a database, triggering payment after the trip is finished, and so on.

Hence, the general formula for a state machine can be given as:

Current State + Some Action / Event= Another State

Let’s see what a state machine designed for a simple video player application would look like:

And we can implement it in code using transitions as follows:

from transitions import Machine

class Video:
   # Define the states
   PLAYING = 'playing'
   PAUSED = 'paused'
   STOPPED = 'stopped'

   def __init__(self, source):
       self.source = source

       # Define the transitions
       transitions = [
           # 1. A video can only be played when it is paused or stopped.
           {'trigger': 'play', 'source': self.PAUSED, 'dest': self.PLAYING},
           {'trigger': 'play', 'source': self.STOPPED, 'dest': self.PLAYING},
           # 2. A video can only be paused when it is playing.
           {'trigger': 'pause', 'source': self.PLAYING, 'dest': self.PAUSED},
           # 3. A video can only be stopped when it is playing or paused.
           {'trigger': 'stop', 'source': self.PLAYING, 'dest': self.STOPPED},
           {'trigger': 'stop', 'source': self.PAUSED, 'dest': self.STOPPED},
       ]

       # Create the state machine
       self.machine = Machine{
           model = self,
           transitions = transitions,
           initial = self.STOPPED
       }

   def play(self):
       pass

   def pause(self):
       pass

   def stop(self):
       pass

Now, in case, we want to add another state, say rewind, we can do that easily as follows:

from transitions import Machine

class Video:
   # Define the states
   PLAYING = 'playing'
   PAUSED = 'paused'
   STOPPED = 'stopped'
   REWINDING = 'rewinding' # new

   def __init__(self, source):
       self.source = source

       # Define the transitions
       transitions = [
           # 1. A video can only be played when it is paused or stopped.
           {'trigger': 'play', 'source': self.PAUSED, 'dest': self.PLAYING},
           {'trigger': 'play', 'source': self.STOPPED, 'dest': self.PLAYING},
           {'trigger': 'play', 'source': self.REWINDING, 'dest': self.PLAYING}, # new
           # 2. A video can only be paused when it is playing.
           {'trigger': 'pause', 'source': self.PLAYING, 'dest': self.PAUSED},
           {'trigger': 'pause', 'source': self.REWINDING, 'dest': self.PAUSED}, # new
           # 3. A video can only be stopped when it is playing or paused.
           {'trigger': 'stop', 'source': self.PLAYING, 'dest': self.STOPPED},
           {'trigger': 'stop', 'source': self.PAUSED, 'dest': self.STOPPED},
           {'trigger': 'stop', 'source': self.REWINDING, 'dest': self.STOPPED}, # new
           # 4. A video can only be rewinded when it is playing or paused.
           {'trigger': 'rewind', 'source': self.PLAYING, 'dest': self.REWINDING}, #new
           {'trigger': 'rewind', 'source': self.PAUSED, 'dest': self.REWINDING}, # new
       ]

       # Create the state machine
       self.machine = Machine{
           model = self,
           transitions = transitions,
           initial = self.STOPPED
       }

   def play(self):
       pass

   def pause(self):
       pass

   def stop(self):
       pass

   def rewind(self):
       pass

Thus, we can see how state machines can simplify a complex implementation and save us from writing incorrect code. Having learned the capabilities of state machines, it’s now important to understand why and when to use state machines.

Why & When to Use State Machines?

State Machines can be utilized in applications that have distinct states. Each stage can lead to one or more subsequent states, as well as end the process flow. A State Machine employs user input or in-state computations to choose which state to enter next.

Many applications necessitate an "initialize" stage, followed by a default state that allows for a wide range of actions. Previous and present inputs, as well as states, can all have an impact on the actions that are executed. Clean-up measures can then be carried out when the system is "shut down."

A state machine can help us conceptualize and manage those units more abstractly if we can break down a hugely complex job into smaller, independent units, where we simply need to describe when a state can transition to another state and what happens when the transition occurs. We don't need to be concerned with how the transition occurs after setup. After that, we only need to think about when and what, not how.

Furthermore, state machines let us see the entire state process in a very predictable way; once transitions are set, we don't have to worry about mismanagement or erroneous state transitions; the improper transition may occur only if the state machine is configured properly. We have a comprehensive view of all states and transitions in a state machine.

If we don't use a state machine, we are either unable to visualize our systems in various possible states, or we are knowingly or unknowingly coupling our components tightly together, or we are writing many if-else conditions to simulate state transitions, which complicates unit and integration testing because we must ensure that all test cases are written to validate the possibility of all the conditions and branching used.

Advantages of State Machines

State machines, in addition to their ability to develop decision-making algorithms, are functional forms of application planning. As applications get more complex, the need for effective design grows.

State diagrams and flowcharts are useful and occasionally essential throughout the design process. State Machines are important not just for application planning but are also simple to create.

Following are some of the major advantages of state machines in modern-day computing:

It helps you eliminate hard coding conditions. On your behalf, the state machine abstracts all logic related to states and transitions.
State machines often have a finite number of states with definite transitions, making it simple to identify which transition/data/event triggered the present state of a request.
After establishing a state machine, developers may concentrate on building actions and preconditions. With sufficient validation and preconditioning, state machines restrict out-of-order operations. As in the Uber example, a driver cannot be rewarded until the voyage is done.
State machines can be quite easy to maintain. The actions taken during each transition are logically independent of one another. As a result, the corresponding code can be isolated.
State machines are less prone to change and are more stable. It becomes much easier to maintain such systems if the current and future use cases are very obvious.

Disadvantages of State Machines

Not everything about state machines is good, they can sometimes lead to drawbacks and challenges too. Here are some of the common problems with state machines:

State machines are usually synchronous. So, if you need asynchronous background API calls/job execution, you'll have to weigh the pros and cons carefully before deciding on the best option.
The code can quickly get jumbled. Because state machines are data-driven, your product team may ask you to execute different transitions from the same state based on different data/input parameters. As a result, this type of demand may result in multiple transitions with a clumsy precondition check. It is entirely dependent on the product and the machine's current configuration.
If you need to load balance state machine instances, go with the one that has persistence enabled; otherwise, you'll need to implement your persistence layer and necessary validations to ensure that multiple requests fired at separate state machine instances produce consistent results.
Because there are few resources or communities dedicated to distinct state machine implementations, assistance may be limited once you've chosen a library.

Things to keep in mind when using State Machines

When using a state machine, your system should ideally have two logical components:

the state machine/workflow system itself
the business logic contained in one or more services.

The state machine may be thought of as the infrastructure that drives state transitions; it verifies state transitions and executes configured actions before, during, and after a transition; however, it should not know what business logic is done in those actions.

So, in general, it's a good idea to isolate the state machine from the core business logic by using correct abstractions; otherwise, managing the code would be a nightmare.

Here are some other real-life scenarios where we need to employ state machine logic with caution:

A state machine is more than just states, transitions, and actions. It should also be capable of defining a border around a state change. A transition can only be successful in particular cases if it is triggered by a trustworthy system or user. There might be a variety of similar situations. As a consequence, we should be able to develop appropriate state transition guarding logic.
We commonly end up with many processes for the same business entity that can run concurrently. In such cases, one process does not obstruct other workflows; they may or may not be triggered concurrently, but they may coexist; the second workflow may begin from one of the first workflow's eligible phases, after which it may branch off and work independently. This sort of use case is established by the business; not every organization will have it.
In principle, workflow systems are independent of the business domain. As a consequence, in the same workflow system, many processes with no link to the same business organization can be established. They may have a shared or distinct starting point, depending on whether the workflow system enables multiple starting points.
When numerous separate workflows are formed in the same workflow system, you get a global picture of all business processes running across various business entities in your system. Depending on the business use cases, different processes may also have certain identical stages.

Practical or real-life use cases for state machines:

Following are some of the practical applications that benefit from the concept of state machines in our daily lives:

Single-page or tabbed dialogue boxes A tab in the conversation box represents each state. A user can initiate a state transition by selecting a certain tab. The status for each tab includes any actions that the user can take.
A self-service banking machine (ATM). In this application, states such as waiting for user input, confirming the needed amount against the account balance, distributing the money, printing the receipt, and so on are all conceivable.
A software that takes a single measurement, stores it in memory, and then waits for the user to take another action. This program's steps include waiting for user input, performing the measurement, recording the data, displaying the results, and so on. Configuring ETL Jobs, for example.
State machines are commonly used for designing user interfaces. While designing a user interface, distinct user actions shift the user interface into separate processing segments. Each of these elements will function as a state in the State Machine. These segments can either lead to another segment for further processing or wait for another user event. In this scenario, the State Machine monitors the user to determine what action they should take next.

When you purchase something from an online e-commerce site, for example, it goes through various phases, such as Ordered, Packed, Shipped, Cancelled, Delivered, Paid, Refunded, and so on. The transition occurs automatically as things move through a warehouse or logistics center and are scanned at various stages, such as when a user cancels or wants a refund.

Process testing is another typical application for State Machines. In this example, each stage of the process is represented by a state. Depending on the results of each state's exam, a separate state may be declared. This might happen frequently if the process under examination is thoroughly studied.

Conclusion

The notion of state machines is extremely useful in programming. It not only streamlines the process of developing more complicated use case apps but also reduces the development work necessary. It provides a more simple and elegant grasp of modern-day events and when correctly applied, may work miracles.

If you find this insightful, do let me know your views in the comments. Also, any kind of feedback is welcome. In case you want to connect with me, follow the links below:

LinkedIn | GitHub | Twitter

HTTP Status Codes That Will Make You Think

Pragati Verma — Sun, 24 Sep 2023 18:15:02 +0000

Last year, I wrote an article on how we might be using some of the HTTP status codes wrong, and this time I discovered some HTTP status codes that we don't use that commonly, but they can have fascinating use cases.

So, in this article, we will discuss some HTTP status codes that will make you think. Let's get started!

It would be more fun if try to guess what the status code means while you read the heading! 😉

103 Early Hints 🫣

The 103 Early Hints HTTP status code is a bit unusual but pretty interesting. It's like a friendly advance notice from a web server to your browser. When a server sends a 103 code, it's saying, "Hey, I'm going to send you a bunch of stuff, and here's a sneak peek to get you started."

It's kind of like when you're at a restaurant, and the chef sends out a little appetizer before your main course. In this case, the server is letting your browser know that more data is on the way, so your browser can start preparing to receive it. It's all about optimizing the loading process and making your web experience smoother.

Its benefits include:

Faster Loading: Provides early hints to browsers, allowing them to start fetching critical resources sooner, leading to faster page rendering.
Enhanced User Experience: Reduces perceived loading times, resulting in a better user experience and higher engagement.
Optimized Resource Fetching: Prioritizes essential assets, optimizing the loading of above-the-fold content.
Reduced Round-Trips: Minimizes the need for additional requests, reducing latency and loading times.
Bandwidth Savings: Avoids fetching unnecessary resources, saving bandwidth for both users and servers.

Read here for more info.

300 Multiple Choices

The HTTP status code 300, "Multiple Choices," is used to indicate that the requested resource has multiple representations available, each with its own specific location (URI), and the user or client must choose one.

When a server returns a 300 status code, it typically includes a "Location" header in the response with a list of URIs for the alternative representations of the resource. The client can then decide which URI to follow based on user preferences or application logic.

For example, if a client requests a resource like a document, and the server has multiple versions of that document in different formats (e.g., HTML, PDF, plain text), it can respond with a 300 status code and provide URIs for each version. The client can then prompt the user to choose the preferred format, or it can use its own logic to make the selection.

The 300 status code is relatively rare in typical web interactions and is mainly used in scenarios where content negotiation is necessary to determine the best representation of a resource based on user or application requirements.

Read here for more info.

303 See Other 👀

The HTTP status code 303 See Other is a redirection status code used to instruct a web client (usually a web browser) to perform a GET request to a different URI. It indicates that the resource the client requested can be found at a different location, and the client should retrieve it from there.

This status code is like a signpost that says, "You asked for something, but it's not here. Go check somewhere else!" It's a way for a web server to tell your browser that the resource you're looking for can be found at a different location or URL.

Wait, doesn't that sound a lot like HTTP status code 302 Found?

HTTP status codes 302 and 303 both deal with redirection, but they serve different purposes. The key distinction lies in the nature of the redirection. The 302 status code indicates a temporary redirection, where the requested resource is temporarily available at a different URI.

In contrast, the 303 status code signifies a more explicit redirection, instructing the client to make a new GET request to the alternative URI for the requested resource.

While 302 implies that future requests may still be directed to the original URI, 303 is often used when you want to ensure that subsequent requests for the same resource always go to the new URI, making it particularly useful for certain web forms and data submissions.

For example, after successfully submitting a form on a website, the server might respond with a 303 code and a "Location" header pointing to a new URI that displays a thank-you message or a confirmation page. This ensures that refreshing the confirmation page won't result in a duplicate form submission. The client is instructed to make a GET request to the new URI to retrieve the result or view the confirmation, decoupling it from the initial POST request.

Read here for more info.

406 Not Acceptable ❌

The HTTP status code 406 Not Acceptable is returned by a web server when a client's request specifies certain response criteria, such as acceptable media types (content types), languages, or character sets, that the server cannot fulfill based on the available resources. In other words, the server cannot provide a response that matches the client's specified preferences.

For example, if a client sends an HTTP request header indicating that it only accepts responses in a particular language or content type, and the server does not have the requested content in that language or format, it would respond with a 406 status code.

The 406 status code is essentially a way for the server to communicate to the client that it cannot meet the specific content requirements specified in the request, and the client may need to adjust its preferences or handle the situation accordingly, such as by requesting a different content type or language.

Sounds a lot like 404 Not Found no? No, it doesn't, if you read it carefully, the 406 status code is about the server having the resource but not in the requested format, while the 404 status code is about the server not finding the resource at all.

Read here for more info.

409 Conflict ⚔️

The HTTP status code 409, "Conflict," is used to indicate that the request could not be completed due to a conflict with the current state of the target resource. This status code typically arises in scenarios involving concurrent updates or conflicting changes to a resource.

For example, if two users attempt to modify the same piece of data simultaneously, a conflict can occur. In this case, the server may respond with a 409 status code to inform the clients that their requests cannot be processed because of the conflicting changes.

The response often includes additional information or guidance on how to resolve the conflict, such as which resource state caused the conflict or how clients can retry their requests after addressing the conflict. It is commonly used in web applications, especially those with collaborative editing or version control features, to manage data consistency in situations where multiple parties can modify the same resource.

Did you think of 400 Bad Request while reading the above case? Well, to clarify that, while a 400 error points to issues with the request itself, a 409 error points to conflicts arising from the state of the resource being modified.

Read here for more info.

410 Gone 🪄

The HTTP status code 410, "Gone," is used to indicate that a previously available resource on the server is no longer accessible and has been intentionally removed or permanently deleted. Unlike the 404 status code, which simply indicates that a resource is not found, the 410 status code explicitly conveys that the resource existed in the past but has now been deliberately taken down, and there is no expectation of it becoming available again in the future.

Website administrators and developers may use the 410 status code when they want to inform both clients and search engines that a resource, such as a webpage or an API endpoint, has been permanently discontinued. This can be useful for maintaining the accuracy of search engine results and informing users that the resource they were seeking is no longer accessible.

Read here for more info.

417 Expectation Failed 😞

The HTTP status code 417, "Expectation Failed," is used to indicate that a server cannot meet the client's expectations specified in the "Expect" request header of the HTTP request.

The "Expect" header is typically used to define certain expectations the client has about how the server should handle the request, such as requesting that the server respond with a specific status code or comply with certain protocol extensions.

When a server encounters an "Expect" header that it cannot fulfill, it responds with a 417 status code. This can happen, for example, if the client expects the server to support a specific authentication method or protocol extension, but the server cannot meet that expectation.

The 417 status code is relatively rare and may not be encountered in everyday web browsing.

Read here for more info.

418 I'm a teapot 🫖

HTTP status code 418, "I'm a teapot," is an April Fools' joke and is not a standard HTTP status code. It was introduced in RFC 2324, a document titled "Hyper Text Coffee Pot Control Protocol," which is not a serious HTTP specification but rather a humorous and playful one.

This status code was intended to be a joke and not meant for actual use in web development or communication between servers and clients. It is not recognized or implemented by web servers or browsers for practical purposes.

So, if you encounter the "418 I'm a teapot" status code in any context, it's likely being used for fun or as a humorous way to communicate that the request is not valid or should not be taken seriously.

Read here for more info.

422 Unprocessable Content 😵‍💫

You might know about the HTTP status code 400, "Bad Request," which indicates that the server cannot process the client's request due to issues with the request itself, such as malformed syntax or missing required parameters. It signifies that the server does not understand or cannot fulfill the request due to client errors.

On the other hand, HTTP status code 422, "Unprocessable Entity," is used when the server can understand the client's request but cannot process it because the request's content, such as data or input, is semantically incorrect, syntactically invalid, or violates validation rules set by the server. It specifically addresses issues with the request's content, making it a more suitable choice when the request entity is problematic but not due to generic syntax errors.

Read here for more info.

451 Unavailable For Legal Reasons 👩‍⚖️

The HTTP status code 451, "Unavailable For Legal Reasons," is used to indicate that a server has received a request for a resource, but it refuses to fulfill the request due to legal restrictions or legal censorship.

This status code is typically used when a government or legal authority has mandated the removal or blocking of specific content, and the server must comply with these legal requirements.

The name "451" is a reference to Ray Bradbury's novel "Fahrenheit 451," in which books are banned and burned by the government. The choice of this status code name symbolizes the idea of content being "unavailable for legal reasons."

When a server responds with a 451 status code, it may also include additional information in the response, such as a description of the legal restrictions or a reference to the relevant laws or regulations.

The 451 status code is relatively recent in the HTTP standard and reflects the growing awareness of legal and censorship-related issues on the internet. Its use is typically associated with cases where access to certain online content is restricted due to legal compliance.

Read here for more info.

Conclusion

There are a lot more HTTP status codes that we don't really use commonly but they exist for specific purposes. I hope that you found this article helpful.

Do let me know if you used any of the above HTTP status codes in any interesting use case.

Keep reading! 👋

Forem: Pragati Verma

[Boost]

Understanding Python's Garbage Collection and Memory Optimization

Pragati Verma ・ Mar 17

Understanding Python's Garbage Collection and Memory Optimization

Memory Organization in Python

Stack Memory

Heap Memory

Python's Primary Memory Management: Reference Counting

Advantages of Reference Counting

Limitations of Reference Counting

Python's Garbage Collection System

Generational Garbage Collection

How Garbage Collection Works Step by Step

Detecting and Breaking Circular References

When Does Garbage Collection Run?

Performance Considerations

Optimizing Memory Management in Python

Avoiding Unnecessary Object Creation

Example: Using Generators Instead of Lists

Handling Circular References Efficiently

Example: Using Weak References to Prevent Cycles

Disabling or Controlling Garbage Collection When Necessary

Example: Disabling Garbage Collection

Monitoring Memory Usage

Example: Using tracemalloc to Track Memory Usage

Real-World Applications

Data Processing Applications

Long-Running Services

Resource-Constrained Environments

Conclusion

Setting Up Apache Kafka Locally: A Step-by-Step Guide

Mastering Apache Kafka: Powering Modern Data Pipelines

Pragati Verma ・ Jan 16

Prerequisites:

Step 1: Install Kafka Using Homebrew

Step 2: Start Zookeeper

Step 3: Start Kafka Server

Step 4: Create a Kafka Topic

Step 5: Produce Messages to the Topic

Step 6: Consume Messages from the Topic

Step 7: Verify Everything is Running

Step 8: Experiment Further

Common Issues and Troubleshooting Tips

1. Kafka Broker Won't Start:

2. Zookeeper Not Starting:

3. Kafka Producer Cannot Connect to Broker:

4. Kafka Consumer Not Receiving Messages:

5. Topic Creation Failures:

6. Kafka Consumer Groups Not Working Properly:

7. Zookeeper and Kafka Version Mismatch:

Conclusion

Are you curious about how modern systems handle massive amounts of real-time data? Apache Kafka is the secret behind seamless data streams, from powering stock market analytics to real-time social media insights. Start your Kafka journey today!

Mastering Apache Kafka: Powering Modern Data Pipelines

Pragati Verma ・ Jan 16

Mastering Apache Kafka: Powering Modern Data Pipelines

What is Apache Kafka?

Key Features of Apache Kafka:

Why Use Apache Kafka?

Core Components of Apache Kafka

1. Topics:

2. Partitions:

3. Producers:

4. Consumers:

5. Brokers:

6. Zookeeper:

Message Flow in Kafka

1. Producers Send Messages to Topics

2. Partitioning and Replication

3. Consumers Read Messages Using Offsets

Real-World Applications of Apache Kafka

Industry Use Cases

Conclusion

Setting Up Apache Kafka Locally: A Step-by-Step Guide

Pragati Verma ・ Jan 21

Scraping Infinite Scroll Pages with a 'Load More' Button: A Step-by-Step Guide

Step 1: Prerequisites

Step 2: Get Access to the Content

Step 3: Load More Products

Step 4: Parse Product Information

Example: Using `tracemalloc` to Track Memory Usage

1. Use `select_related()` and `prefetch_related()`

2. Use `annotate()` to perform calculations

3. Use `Subquery` for complex calculations