Forem: Chandrashekhar Kachawa

System Design Fundamentals: From Monolith to Microservices

Chandrashekhar Kachawa — Wed, 07 Jan 2026 09:11:37 +0000

Choosing the right architecture is one of the most critical decisions you'll make when building a software application. It impacts how quickly you can ship features, how well your application scales, and how easy it is to maintain. Two of the most talked-about architectural patterns are the monolith and microservices.

This isn't a battle of which is "better," but a discussion of trade-offs. Let's break down what each pattern means and explore the journey from one to the other.

The Monolith: The All-in-One Application

A monolithic architecture is the traditional way of building applications. It's a single, unified codebase where all the application's components—UI, business logic, data access layer—are developed, deployed, and scaled as a single unit.

Imagine a large retail store. Everything is in one massive building: clothing, electronics, groceries, and the checkout counters. This is a monolith.

Pros of a Monolith

Simplicity of Development: When you're starting, having everything in one codebase is straightforward. There's no complex distributed system to worry about.
Easy to Test: End-to-end testing is simpler because you're testing a single, self-contained application.
Single Deployment: You deploy one unit. This simplifies your CI/CD pipeline initially.
Less Overhead: No need to manage communication between different services.

Cons of a Monolith

Tight Coupling: As the application grows, components become tightly coupled and hard to isolate. A change in one part can have unintended consequences in another.
Difficult to Scale: You have to scale the entire application, even if only one small part is a bottleneck. You can't scale the user authentication service independently of the image processing service.
Technology Stack Lock-in: You're committed to a single technology stack. Adopting a new language or framework for a specific feature is difficult.
Slowing Development Velocity: As the codebase gets larger, it becomes harder to understand, compile times increase, and developer onboarding slows down.

Microservices: A Constellation of Services

A microservice architecture breaks down an application into a collection of smaller, independent services. Each service is responsible for a specific business capability, runs in its own process, and communicates with other services over a network, typically via APIs.

Think of it as a city. Instead of one giant building, you have a specialized hospital, a power plant, a water treatment facility, and a telecommunications tower. Each is independent, can be upgraded separately, but they all work together.

Pros of Microservices

Independent Scaling: You can scale individual services based on their specific needs. If your video encoding service is under heavy load, you can scale just that service.
Technology Freedom: Each service can be built with the best technology for its job. You could have a service in Python for machine learning, another in Go for high-concurrency tasks, and another in Node.js for its API gateway.
Fault Isolation: If one service fails, it doesn't necessarily bring down the entire application.
Parallel Development: Teams can develop, deploy, and manage their services independently, leading to faster development cycles in large organizations.

Cons of Microservices

Operational Complexity: You're now managing a distributed system. This means more moving parts, more deployments, and a need for robust automation and monitoring.
Network Latency: Communication between services over a network is slower than in-process calls within a monolith.
Data Consistency: Maintaining data consistency across multiple services is a significant challenge. This often requires complex patterns like sagas or eventual consistency.
Distributed Debugging: Debugging a request that spans multiple services is significantly harder. Centralized logging and tracing become essential.

The Migration: When and How?

Very few applications should start with microservices. The operational overhead is often too high for a new product. Most successful microservice architectures begin life as a monolith that became successful enough to warrant a migration.

So, when do you consider migrating?

Scaling Bottlenecks: You find that one part of your monolith is consuming most of the resources, and you need to scale it independently.
Development Slowdown: Your teams are stepping on each other's toes, and deploying new features has become slow and risky.
Need for Specialized Technology: A new business requirement is best served by a technology that doesn't fit your monolithic stack.

The Strangler Fig Pattern

A "big bang" rewrite is almost always a bad idea. A proven, incremental approach is the Strangler Fig Pattern.

Identify a Seam: Find a distinct piece of functionality in your monolith that you can "strangle" (e.g., user profile management).
Build a New Service: Create a new microservice that implements this functionality.
Redirect Traffic: Introduce a proxy or an API gateway that sits in front of the monolith. Initially, it just passes all traffic to the monolith. Then, you configure it to route traffic for the specific functionality (e.g., /api/users/*) to your new microservice. The old monolithic code is still there, but it's no longer receiving traffic.
Repeat: Continue this process, gradually building new services and routing traffic away from the monolith. Over time, the monolith shrinks and is eventually "strangled" by the new microservices.

Conclusion

The choice between a monolith and microservices is about understanding trade-offs and choosing the right tool for the job at the right time.

Start with a monolith. It's simpler and allows you to move quickly.
Keep it modular. Even within a monolith, practice good software design. Keep your code organized and your components loosely coupled. This will make a future migration much easier.
Migrate when the pain is real. Don't jump to microservices because it's trendy. Do it when the limitations of your monolith are actively hindering your ability to scale your application or your team.

By understanding these fundamentals, you can make informed architectural decisions that set your application up for long-term success.

How to Design APIs

Chandrashekhar Kachawa — Fri, 02 Jan 2026 07:14:38 +0000

As developers, we're often taught to build REST APIs following a standard set of conventions. We create multiple endpoints, use various HTTP verbs, and return a range of status codes like 200, 404, or 500. But what if this approach has fundamental flaws, especially for private, internal APIs?

A senior engineer's perspective often challenges these conventions in favor of a more robust, predictable, and proprietary architecture. The goal is to build APIs that are not just functional, but also highly scalable, secure, and efficient. Let's explore how to achieve this.

Core Architecture: A New Paradigm

The foundation of a senior-level API design moves away from the classic REST model and embraces a more consolidated and efficient structure.

1. Move Away from REST for Private APIs

Standard REST status codes can be misleading. A 500 error might hide a predictable validation failure, and a 404 doesn't distinguish between a mistyped URL and a resource that genuinely doesn't exist. For private APIs (those consumed by your own front end), we can do better. A proprietary design gives us more control and clarity.

2. Embrace a Single Endpoint and Batching

Instead of a sprawling collection of URLs (/users, /posts, /comments), consider using a single /api endpoint that only accepts POST requests. This might sound counterintuitive, but it unlocks a powerful capability: request batching.

Client applications can collect multiple actions (e.g., fetching user data, posting a comment, updating a setting) and send them to the server in a single network call. By batching requests at short intervals (e.g., every 50ms), you dramatically reduce network overhead and improve perceived performance.

3. A Clean, Folder-Based Structure

To keep the codebase organized, each API should live in its own folder. This self-contained module should include at least two key files:

handler.ts: The file containing the core execution logic for the API.
constants.ts: A metadata file that defines the API's behavior, such as its rate-limiting configuration and, most importantly, its validation schemas.

The Request Flow: Security and Predictability First

A senior engineer knows that an API's primary responsibility is to be secure and predictable. This is achieved by establishing a strict, ordered flow for every incoming request.

1. Validation: The Most Critical Step

Before your handler's logic ever runs, you must strictly validate the request body. This is non-negotiable. Tools like Zod are perfect for this, allowing you to define a precise schema for the expected input.

// src/content/blog/api/some-api/constants.ts
import { z } from 'zod';

export const SomeApiInputSchema = z.object({
  userId: z.string().uuid(),
  content: z.string().min(1).max(280),
});

Strict validation is your first line of defense against a host of vulnerabilities, including NoSQL injections. If the incoming data doesn't match the schema, the request is rejected immediately. No exceptions.

2. Smart Rate Limiting

Rate limiting prevents abuse and ensures stability. The strategy should differ based on who is making the request:

Unauthorized Users: Rate limit based on their IP address.
Authorized Users: Rate limit based on their Account ID and the specific API they are calling. This is more granular and fair.

Pro-tip: You can embed the AccountID in a JWT-signed token. This allows your rate-limiting middleware to extract the ID without needing a costly database lookup, making the process incredibly fast.

3. Authorization: The First Line of Code

Security shouldn't be an afterthought; it should be the very first thing you check. Every API handler should begin with an explicit authorization check.

// src/content/blog/api/some-api/handler.ts
import { requireAdmin } from '~/lib/auth';

export async function handler(input: SomeApiInput) {
  requireAdmin(input.currentUser); // Throws an error if not an admin

  // ... rest of the logic
}

This requireAdmin or requireUser function acts as a gatekeeper, ensuring that the rest of your code only executes if the user has the necessary permissions.

End-to-End Type Safety: The Holy Grail

While input schemas provide security, output schemas provide type safety and a superior developer experience. By defining a Zod schema for the data your API returns, you create a contract.

This contract can then be shared with your front-end application. Using a type generator, you can create a fully typed, end-to-end system. When the back end changes an output shape, the front-end code will immediately show a TypeScript error, catching bugs at compile time, not in production.

Trade-Offs and Considerations

This opinionated design is powerful, but it's not a silver bullet. It's crucial to understand the trade-offs.

Pros

Improved Performance: Request batching dramatically reduces network latency for chatty applications.
Enhanced Developer Experience: End-to-end type safety catches bugs early and makes development a breeze.
Superior Security: A validation-first and authorization-first approach hardens your API from the ground up.

Cons

Breaks REST Conventions: This is the biggest drawback. You lose standard HTTP caching, semantic verb meanings (GET, PUT), and browser-friendly testing.
Poor for Public APIs: This pattern is ill-suited for third-party developers who expect standard RESTful interfaces.
Tooling Complexity: Standard API tools like Swagger or Postman are built for REST and may not integrate seamlessly.

This architecture, similar to what libraries like tRPC popularize, is an excellent choice for modern, tightly-coupled applications where you control both the client and server. For public-facing APIs, sticking to REST is often the better choice.

Conclusion

Designing APIs like a senior engineer means prioritizing predictability, security, and performance. By moving away from traditional REST for private services and embracing concepts like single endpoints, request batching, strict validation, and end-to-end type safety, you can build a back-end architecture that is not only robust and scalable but also a joy to work with.

A Practical Guide to Backing Up Your Linux Server: PostgreSQL, Jenkins, and More

Chandrashekhar Kachawa — Wed, 31 Dec 2025 06:41:41 +0000

As developers and system administrators, we build and manage services that are critical to our applications. But what happens when disaster strikes? A hardware failure, a software bug, or even human error can lead to catastrophic data loss. This is where a robust backup strategy isn't just a good idea—it's a necessity.

This guide provides a practical, hands-on approach to backing up some of the most common and critical services running on a Linux server: PostgreSQL databases, Jenkins automation servers, and other essential system files.

1. Backing Up Your PostgreSQL Database

Your database is often the most critical component of your application stack. pg_dump is the standard, powerful utility for creating consistent backups of a PostgreSQL database.

On the Server: Creating the Backup

The pg_dump command connects to your database and writes its contents to a file. You can run this directly from your server's command line.

A basic command looks like this:

pg_dump -U your_user -d your_db -f backup.sql

-U your_user: The PostgreSQL user to connect as.
-d your_db: The name of the database to back up.
-f backup.sql: The output file for the backup.

For more flexibility during restoration, it's highly recommended to use the custom format (-F c):

pg_dump -U your_user -d your_db -F c -f backup.dump

This format is compressed and allows for more advanced restoration options with the pg_restore utility, such as reordering or restoring specific objects.

Downloading the Backup to Your Local PC

Once you have the backup file on your server, you need a secure way to download it to your local machine. The scp (Secure Copy) command is perfect for this, as it transfers files over an encrypted SSH connection.

scp user@your_server_ip:/path/to/backup.dump .

user@your_server_ip: Your username and the IP address of your server.
/path/to/backup.dump: The full path to the backup file on the server.
.: The destination on your local machine (the current directory).

Pro Tip: Using a Specific SSH Key

If your server requires a specific SSH key for authentication, you can provide it using the -i (identity file) flag. Remember to ensure your key file has secure permissions (chmod 600 /path/to/key).
scp -i /path/to/your/private_key user@your_server_ip:/path/to/backup.dump .
For an even cleaner workflow, consider adding an entry to your SSH config file (~/.ssh/config):
Host my-prod-server
    HostName your_server_ip
    User user
    IdentityFile /path/to/your/private_key
With this config, your command simplifies to scp my-prod-server:/path/to/backup.dump ..

2. Backing Up Your Jenkins Server

Jenkins stores all its data—including job configurations, build history, plugins, and system settings—in a single directory known as JENKINS_HOME. Backing up Jenkins is as simple as creating an archive of this directory.

First, find your JENKINS_HOME directory. On most Linux systems, it's located at /var/lib/jenkins.

You can use the tar utility to create a compressed archive:

tar -czvf jenkins_backup_$(date +%F).tar.gz /var/lib/jenkins

-c: Create a new archive.
-z: Compress the archive with gzip.
-v: Verbosely list the files processed.
-f: Specifies the output archive file.
$(date +%F): This handy command substitution appends the current date (YYYY-MM-DD) to the filename.

Just like with the database backup, you can use scp to download this jenkins_backup.tar.gz file to your local machine.

3. Backing Up Essential System Files

Beyond specific services, your server has other critical files, especially in /etc (system-wide configurations) and /home (user data). You can use tar to back these up as well.

tar -czvf system_backup_$(date +%F).tar.gz /etc /home /var/www

This command archives the configuration directory, all user home directories, and a common web root directory (/var/www).

4. Automating Your Backups with Cron

Manual backups are better than no backups, but automated backups are what give you true peace of mind. On Linux, the cron daemon is the standard way to schedule recurring tasks.

First, create a simple shell script to perform the backups.

backup_script.sh

#!/bin/bash

# Set backup directory
BACKUP_DIR="/opt/backups"
mkdir -p $BACKUP_DIR

# Timestamp
TIMESTAMP=$(date +%F)

# Backup PostgreSQL
pg_dump -U your_user -d your_db -F c -f $BACKUP_DIR/db_backup_$TIMESTAMP.dump

# Backup Jenkins
tar -czf $BACKUP_DIR/jenkins_backup_$TIMESTAMP.tar.gz /var/lib/jenkins

# Backup system files
tar -czf $BACKUP_DIR/system_backup_$TIMESTAMP.tar.gz /etc /home

Make the script executable: chmod +x backup_script.sh.

Now, open your crontab to edit it:

crontab -e

Add a line to schedule the script. For example, to run it every day at 2:00 AM:

0 2 * * * /path/to/your/backup_script.sh

Conclusion

Regularly backing up your services is one of the most important responsibilities of managing a server. With tools like pg_dump, tar, scp, and cron, you can create a powerful, automated, and secure backup strategy.

But remember the golden rule: a backup is worthless until you have successfully tested a restoration. Periodically practice restoring your backups to a staging environment to ensure they work when you need them most.

Mastering Real-Time Communication with Socket.IO Rooms

Chandrashekhar Kachawa — Mon, 22 Dec 2025 04:57:57 +0000

Socket.IO is a powerful library that enables real-time, bidirectional, and event-based communication between web clients and servers. While broadcasting messages to all connected clients is simple, most real-world applications require sending messages to specific groups of users. This is where Socket.IO's "rooms" feature shines.

This guide will walk you through setting up and using rooms in Node.js, covering everything from creation to managing public and private spaces.

What are Rooms?

A room is a server-side concept that allows you to group sockets together. Sockets can join and leave rooms, and you can broadcast messages to all sockets within a specific room. A single socket can be in multiple rooms at once.

Every socket automatically joins a room identified by its own unique socket.id. This is useful for sending private messages to a specific user.

Setting Up the Basic Server

First, let's set up a basic Node.js server with Express and Socket.IO.

pnpm install express socket.io

Now, create your main server file (e.g., server.js):

// server.js
import express from 'express';
import { createServer } from 'http';
import { Server } from 'socket.io';

const app = express();
const httpServer = createServer(app);
const io = new Server(httpServer, {
  cors: {
    origin: "*", // Allow all origins for simplicity
  },
});

io.on('connection', (socket) => {
  console.log(`A user connected: ${socket.id}`);

  // Listen for a custom event to join a room
  socket.on('joinRoom', (roomName) => {
    socket.join(roomName);
    console.log(`${socket.id} joined room: ${roomName}`);

    // Broadcast to the specific room
    io.to(roomName).emit('message', `Welcome ${socket.id} to the ${roomName} room!`);
  });

  socket.on('disconnect', () => {
    console.log(`A user disconnected: ${socket.id}`);
  });
});

const PORT = process.env.PORT || 3000;
httpServer.listen(PORT, () => {
  console.log(`Server is running on port ${PORT}`);
});

Room Creation and Uniqueness

You might have noticed we never explicitly "created" a room. Rooms are created implicitly when the first socket joins them. They are automatically disposed of when the last socket leaves.

The uniqueness of a room is simply based on its name (a string). It's your application's responsibility to manage these names.

How to Ensure Unique Rooms

For Public Rooms: Use predefined, human-readable names like general, support, or news.
For Private Rooms (e.g., Direct Messages): You need a consistent and unique identifier for the two users involved. A common strategy is to combine their unique user IDs in a predictable order.

For example, if you have two user IDs, userA_id and userB_id:

function getPrivateRoomName(id1, id2) {
  // Sort the IDs to ensure the room name is always the same
  // regardless of who initiates the chat.
  return [id1, id2].sort().join('-');
}

const roomName = getPrivateRoomName('userA_id', 'userB_id'); // "userA_id-userB_id"
socket.join(roomName);

This guarantees that any two users will always share the same private room.

Public vs. Private Rooms

The distinction between public and private rooms isn't a feature of Socket.IO itself but a pattern implemented in your application's logic.

Public Rooms

A public room is one that any user can join. The client-side code would typically have a list of available public rooms, and the user can choose one to join.

// Client-side code
const socket = io("http://localhost:3000");

// User joins the 'general' chat room
socket.emit('joinRoom', 'general');

socket.on('message', (data) => {
  console.log(data);
});

Private Rooms

A private room has controlled access. You should always validate on the server whether a user is authorized to join a specific room. This is crucial for security.

// Server-side code
socket.on('joinPrivateRoom', ({ roomId }) => {
  // Example: Check if the user is authenticated and has permission
  const isAuthorized = checkUserAuthorization(socket.request.user, roomId);

  if (isAuthorized) {
    socket.join(roomId);
    io.to(roomId).emit('message', `${socket.id} has joined the private discussion.`);
  } else {
    socket.emit('error', 'You are not authorized to join this room.');
  }
});

When to Use Socket.IO Rooms

Chat Applications: The most obvious use case. Create rooms for group chats, direct messages, or topic-based discussions.
Real-Time Collaboration: Think Google Docs. Each document can be a room, and all edits are broadcast only to the users currently viewing that document.
Multiplayer Games: Group players in a game session into a room. Game state updates are sent only to players in that session, not everyone on the server.
Notification Systems: When a user performs an action, you can send a notification to a room containing their teammates or followers.

When NOT to Use Rooms

Simple Data Fetching: If a client just needs to pull data from the server, a standard REST or GraphQL API is more appropriate. WebSockets add unnecessary complexity.
Broadcasting to Everyone: If you want to send a message to every single connected client, you don't need a room. Just use io.emit().
Unidirectional Updates: If you only need to send data from the server to the client (e.g., stock tickers, news feeds), consider Server-Sent Events (SSE). SSE is a simpler protocol built on top of HTTP and is often sufficient for these scenarios.

Conclusion

Socket.IO rooms are a fundamental concept for building scalable, real-time applications. By grouping sockets, you can efficiently manage communication and ensure that messages are delivered only to the intended recipients. The key is to remember that rooms are a server-side abstraction, and their "privacy" or "publicity" is determined entirely by your application's logic and authorization checks.

Hoppscotch: The Modern, Lightweight Alternative to Postman and Bruno

Chandrashekhar Kachawa — Thu, 18 Dec 2025 10:30:12 +0000

In the world of software development, API client tools are as essential as a good code editor. For years, Postman has been the reigning champion, the default choice for developers everywhere. More recently, tools like Bruno have emerged, catering to developers who prefer a more git-native, offline-first approach. But what if there was a tool that combined the best of both worlds—a fast, web-native experience with the flexibility of open-source?

Enter Hoppscotch, the open-source API development suite that’s rapidly gaining popularity. It’s a powerful, lightweight, and beautifully designed tool that challenges the way we think about API testing and collaboration.

What is Hoppscotch?

At its core, Hoppscotch is a web-based API client. You can simply navigate to their website and start making requests immediately—no installation required. But don't let its simplicity fool you. It's a feature-rich platform that supports REST, GraphQL, and real-time protocols like WebSocket, Socket.IO, and MQTT.

Being open-source, it offers a level of transparency and community involvement that proprietary tools can't match. Plus, for teams concerned with data privacy, it can be entirely self-hosted.

Hoppscotch vs. Postman: A Battle of Philosophies

Postman is a powerful, mature platform, but its recent focus on cloud features and team collaboration has come at a cost. The desktop application can feel heavy, slow to start, and many of its most compelling features are locked behind a paid subscription and require an account.

Here’s how Hoppscotch offers a refreshing alternative:

Performance and Accessibility: Hoppscotch is web-first, meaning it's incredibly fast and accessible from any device with a browser. There's no hefty installation, and it launches instantly. For those who prefer a desktop experience, it's available as a Progressive Web App (PWA).
True Open Source and Cost-Effectiveness: Hoppscotch is free and open-source. While they offer a paid cloud service for teams, the core functionality is available to everyone. Postman's free tier has become increasingly restrictive, pushing users towards paid plans for what many consider basic features.
Collaboration Model: Hoppscotch allows for creating team workspaces and sharing collections without forcing every user into a complex, account-based ecosystem. You can instantly share a request via a link. Postman’s collaboration is powerful but tightly integrated with its paid cloud services.
User Experience: Hoppscotch boasts a clean, modern, and intuitive UI that feels less cluttered than Postman's feature-packed interface.

For developers tired of Postman's sluggish performance and account-walled garden, Hoppscotch is a breath of fresh air.

Hoppscotch vs. Bruno: A Different Kind of Open Source

Bruno is another fantastic open-source tool that has gained a following for its unique, git-friendly approach. It stores all collection data directly on your local filesystem, allowing you to version control your API tests just like your code. This is a brilliant concept, but it serves a different workflow compared to Hoppscotch.

Philosophy: Hoppscotch is web-first, designed for speed and seamless online access. Bruno is local-first, designed for developers who want to manage everything within their Git repository.
Collaboration: Collaboration in Bruno happens via pull requests, which is perfect for teams deeply embedded in a GitOps workflow. Hoppscotch offers more immediate, real-time collaboration through shared workspaces and links, which is often faster for quick debugging or pairing sessions.
Accessibility: With Hoppscotch, you can send a teammate a URL, and they can immediately see and interact with the request. With Bruno, your teammate needs to have the app installed and pull the latest changes from the repository.

If your team's identity is "everything-in-Git," Bruno is a perfect fit. If your team values speed, accessibility, and real-time web-based tools, Hoppscotch has the edge.

Key Features That Make Hoppscotch Shine

Beyond the comparisons, Hoppscotch has a feature set that makes it a compelling tool on its own:

Multi-Protocol Support: It's not just for REST. First-class support for GraphQL (with a schema explorer) and real-time protocols is a huge advantage.
Pre-Request Scripts & Tests: Write JavaScript code to run before a request is sent or to validate the response, similar to Postman.
Environments & Variables: Manage different environments (like development, staging, production) with ease.
PWA for Offline Use: Install Hoppscotch as a PWA to get a desktop-like experience and use it even when you're offline.
Self-Hosting: For ultimate data privacy and control, you can host your own instance of Hoppscotch with just a few commands.

Conclusion

The choice of an API client depends heavily on your team's workflow and priorities.

Postman remains a powerful, enterprise-grade tool, but its performance and cost can be significant drawbacks.
Bruno is the ideal choice for developers who want to live entirely within their Git workflow, treating API collections as code.
Hoppscotch carves out a unique and compelling niche. It's the perfect tool for developers and teams who prioritize speed, accessibility, and a clean user experience, all wrapped in a transparent, open-source package.

If you haven't tried it yet, you owe it to yourself to give Hoppscotch a spin. It might just become your new default API client.

Mastering Python Monorepos: A Practical Guide

Chandrashekhar Kachawa — Fri, 12 Dec 2025 10:33:08 +0000

As projects grow, managing dependencies and shared code across multiple repositories can become a significant challenge. Code gets duplicated, dependency versions drift apart, and coordinating changes becomes a complex dance. A monorepo—a single repository containing multiple distinct projects—offers a powerful solution to these problems.

This guide will walk you through building a scalable Python monorepo, integrating different services like FastAPI and Apache Airflow, and maintaining high code quality with modern tools like Ruff.

Why a Monorepo?

Before we dive in, let's clarify the benefits:

Simplified Dependency Management: A single set of dependencies at the root level (or well-defined per-project dependencies) prevents version conflicts.
Atomic Commits: Changes across multiple services can be made in a single commit, ensuring consistency and simplifying rollbacks.
Seamless Code Sharing: Reusing code between projects is as simple as a local import, eliminating the need for a separate package registry.
Unified Tooling: A single configuration for linting, formatting, and testing can be enforced across the entire codebase.

Designing the Monorepo Structure

A clean structure is key to a maintainable monorepo. Here’s a proven layout:

my-monorepo/
├── .gitignore
├── pyproject.toml      # Root config for dependencies and tools (Ruff)
├── packages/
│   └── shared_library/
│       ├── pyproject.toml  # Makes this an installable package
│       └── src/
│           └── shared_library/
│               ├── __init__.py
│               └── core.py       # Shared business logic
└── services/
    ├── fastapi_app/
    │   ├── pyproject.toml
    │   └── src/
    │       └── fastapi_app/
    │           └── main.py
    └── airflow_dags/
        ├── pyproject.toml
        └── src/
            └── airflow_dags/
                └── example_dag.py

packages/: Contains local Python packages intended to be shared across different services.
services/: Contains the actual applications, like your FastAPI microservice or Airflow DAGs.
pyproject.toml: The root pyproject.toml is crucial for centralizing tool configuration and, optionally, base dependencies.

Creating a Shared Package

The magic of a monorepo lies in its ability to share code effortlessly. Let's make shared_library an installable package.

packages/shared_library/pyproject.toml

[project]
name = "shared_library"
version = "0.1.0"

packages/shared_library/src/shared_library/core.py

# Example shared function
def get_greeting() -> str:
    return "Hello from the shared library!"

Now, any service in your monorepo can install this package in editable mode. This means changes to the library code are immediately reflected in the services that use it, without needing a re-installation.

From the root of the monorepo, you would typically install this into the virtual environment of a service that needs it:

# Example: Installing for the fastapi_app
pip install -e packages/shared_library

Integrating Your Services

Let's see how our FastAPI app and Airflow DAGs can use the shared_library.

1. FastAPI Microservice

Your FastAPI app can now import directly from the shared code.

services/fastapi_app/src/fastapi_app/main.py

from fastapi import FastAPI
from shared_library.core import get_greeting

app = FastAPI()

@app.get("/")
def read_root():
    greeting = get_greeting()
    return {"message": greeting}

When you run this FastAPI application, it will execute the code from shared_library as if it were a standard installed package.

2. Apache Airflow DAGs

Similarly, your Airflow DAGs can pull in shared logic for tasks, connections, or configurations.

services/airflow_dags/src/airflow_dags/example_dag.py

from __future__ import annotations

import pendulum

from airflow.models.dag import DAG
from airflow.operators.python import PythonOperator
from shared_library.core import get_greeting

def print_greeting():
    """A simple Python function to be executed by Airflow."""
    message = get_greeting()
    print(message)

with DAG(
    dag_id="example_shared_library_dag",
    start_date=pendulum.datetime(2023, 1, 1, tz="UTC"),
    catchup=False,
    schedule=None,
    tags=["example"],
) as dag:
    PythonOperator(
        task_id="print_shared_greeting",
        python_callable=print_greeting,
    )

For Airflow to find your shared_library, you must ensure it's installed in the Python environment where your Airflow scheduler and workers are running.

Tooling and Best Practices with Ruff

A monorepo is the perfect environment for standardized tooling. Ruff is an incredibly fast linter and formatter that can be configured at the root of your project.

In your root pyproject.toml, add the following configuration:

[tool.ruff]
# Enable Pyflakes, Pycodestyle, and isort rules
select = ["E", "F", "I"]
line-length = 88

[tool.ruff.lint]
# Define source directories for Ruff to check
include = [
    "packages/shared_library/src/**/*.py",
    "services/fastapi_app/src/**/*.py",
    "services/airflow_dags/src/**/*.py",
]

[tool.ruff.format]
# Use black-compatible formatting
quote-style = "double"

Now, you can run Ruff from the root of your monorepo to check and format your entire codebase with a single command:

# Check for linting errors
ruff check .

# Format all files
ruff format .

This ensures that all your Python code, regardless of which service it belongs to, adheres to the same quality standards.

Conclusion

Adopting a Python monorepo requires a shift in mindset but offers immense rewards in terms of maintainability, consistency, and development speed. By creating a logical structure with shared packages and enforcing unified tooling with Ruff, you can build a robust foundation that scales with your projects. This architecture empowers you to manage complex systems like co-dependent microservices and data pipelines with greater confidence and efficiency.

Demystifying Retrieval-Augmented Generation (RAG)

Chandrashekhar Kachawa — Thu, 11 Dec 2025 07:47:38 +0000

Large Language Models (LLMs) have revolutionized the way we interact with information, but they have a fundamental limitation: their knowledge is frozen at the time they were trained. They can't access real-time information and can sometimes "hallucinate" or invent facts. This is where Retrieval-Augmented Generation (RAG) comes in.

What is RAG?

RAG is a technique that enhances the capabilities of LLMs by connecting them to external knowledge bases. In simple terms, instead of relying solely on its pre-trained data, the model first retrieves relevant information from a specific dataset (like your company's internal documents, a database, or a specific website) and then uses this information to generate a more accurate and context-aware response.

It’s like an open-book exam for an LLM. The model doesn't have to memorize everything; it just needs to know how to look up the right information before answering.

Why Do We Need It?

The primary motivation for RAG is to overcome the inherent limitations of standalone LLMs:

Knowledge Cutoffs: LLMs are unaware of events or data created after their training. RAG provides a direct line to up-to-the-minute information.
Hallucinations: When an LLM doesn't know an answer, it might generate a plausible-sounding but incorrect response. RAG grounds the model in factual data, significantly reducing the chances of hallucination.
Lack of Specificity: A general-purpose LLM knows a lot about everything but may lack deep knowledge of a specific, niche domain (like your organization's internal policies or a technical knowledge base). RAG allows you to inject this domain-specific expertise.
Verifiability: With RAG, you can often cite the sources used to generate the answer, providing a way for users to verify the information.

How Does RAG Work?

The RAG process can be broken down into a few key steps. Let's imagine we're building a chatbot to answer questions based on a set of internal documents.

Step 1: Indexing (The Setup Phase)

Before you can retrieve anything, you need to prepare your knowledge base.

Load Documents: Your documents (e.g., PDFs, Markdown files, database records) are loaded.
Chunking: Each document is broken down into smaller, manageable chunks of text.
Embedding: Each chunk is converted into a numerical representation (a vector embedding) using an embedding model. These embeddings capture the semantic meaning of the text.
Storing: The chunks and their corresponding embeddings are stored in a specialized database called a vector store, which is optimized for fast similarity searches.

This indexing process is typically done offline and only needs to be repeated when your source documents change.

Step 2: Retrieval and Generation (The Live Phase)

This is what happens every time a user asks a question.

User Query: The user submits a query (e.g., "What is our policy on remote work?").
Embed Query: The user's query is also converted into a vector embedding using the same model from the indexing phase.
Search: The vector store is searched to find the text chunks whose embeddings are most similar to the query's embedding. These are the most relevant pieces of information from your knowledge base.
Augment: The original query and the retrieved text chunks are combined into a single, enriched prompt.
Generate: This augmented prompt is sent to the LLM, which then generates a final answer based on the provided context.

Conceptual Python Example

Here’s a simplified Python-like pseudocode to illustrate the live phase:

# Assume we have these pre-configured components:
# - vector_store: A database of our indexed document chunks
# - embedding_model: A model to convert text to vectors
# - llm: A large language model for generation

def answer_question_with_rag(query: str) -> str:
    """
    Answers a user's query using the RAG process.
    """
    # 1. Embed the user's query
    query_embedding = embedding_model.embed(query)

    # 2. Retrieve relevant context from the vector store
    relevant_chunks = vector_store.find_similar(query_embedding, top_k=3)

    # 3. Augment the prompt
    context = "\n".join(relevant_chunks)
    augmented_prompt = f"""
    Based on the following context, please answer the user's query.
    If the context does not contain the answer, say so.

    Context:
    {context}

    Query:
    {query}
    """

    # 4. Generate the final answer
    final_answer = llm.generate(augmented_prompt)

    return final_answer

# Example usage:
user_query = "What is our policy on remote work?"
response = answer_question_with_rag(user_query)
print(response)

When to Use RAG

RAG is incredibly powerful for specific use cases:

Customer Support Chatbots: Provide answers based on product manuals, FAQs, and historical support tickets.
Internal Knowledge Bases: Allow employees to ask questions about company policies, technical documentation, or project histories.
Personalized Content Recommendation: Recommend articles or products based on a user's query and a catalog of items.
Educational Tools: Create "ask the book" or "ask the lecture" applications where students can query course materials.

When Not to Use RAG

RAG is not a silver bullet. There are times when it's unnecessary or might even be counterproductive:

Highly Creative or Open-Ended Tasks: If you want the LLM to write a poem, a fictional story, or brainstorm ideas without constraints, feeding it specific documents is unnecessary.
General Knowledge Questions: For questions like "What is the capital of France?", the LLM's internal knowledge is more than sufficient and much faster to access.
When Latency is Extremely Critical: The retrieval step adds a small amount of latency. If you need an answer in milliseconds and the query is simple, a direct LLM call might be better.
Simple Command-and-Control: For tasks like "turn on the lights" or "play music," a simpler NLU (Natural Language Understanding) system is more appropriate than a full RAG pipeline.

By understanding its strengths and limitations, you can leverage RAG to build more accurate, reliable, and useful AI-powered applications.

Life Update #1

Chandrashekhar Kachawa — Mon, 08 Dec 2025 14:07:04 +0000

It feels like I'm navigating through a thick fog with no compass. I am really feeling lost, unworthy, and like I can't win. Every day feels like a battle, and lately, I've been losing every single one.

A significant part of this weight comes from my career, or the lack thereof. I have been searching for a job for quite a while now, and the process has been soul-crushing. I pour hours into applications, tailoring my resume, and writing cover letters, only to be met with silence or the cold sting of automated rejection emails. It makes me question everything I thought I was good at. I have these skills, this experience, but it feels like none of it matters. I hate that I cannot find a decent job for what I can do.

This professional struggle bleeds into everything else. Neither is life going well. It feels like I'm stuck in place while the world moves on around me. The frustration and disappointment have curdled into a sharp self-loathing. I just very, very dearly hate myself now. It's a constant, nagging voice in my head that tells me I'm not good enough, that I'm a failure.

And in the midst of this turmoil, my heart aches with a grief that feels as fresh as ever. What specially bothers me is my mom. She appears in my dreams, vivid and real. For a fleeting moment, she's there, and everything feels okay. Then I wake up, and the emptiness rushes back in, a painful reminder of what I've lost. I miss her the most. Her absence is a void that nothing can fill.

These dreams make me think about my life and my role in it. I wish I was a good enough son. I carry this heavy feeling that I've let her down, that I haven't lived up to the hopes she had for me.

I feel like I've failed at everything—my career, my personal life, my duties as a son. The path forward is completely obscured. I am not sure what I should even do now. The weight of it all is immense.

Even writing this fills my eyes with tears. But maybe, just maybe, putting it into words is a start.

From npm to uv: A Node.js Developer's Guide to Python Project Setup

Chandrashekhar Kachawa — Fri, 05 Dec 2025 08:26:36 +0000

If you're a Node.js developer stepping into the Python world, you're probably used to the speed and simplicity of tools like npm, pnpm, or yarn. You run npm install, your dependencies pop into node_modules, and you're off to the races. Then you encounter Python and hear about pip, venv, virtualenv, pipenv, poetry... and it all feels a bit... fragmented.

What if I told you there's a tool that brings a familiar, unified, and blazingly fast experience to Python, much like the modern tools in the JavaScript ecosystem?

Enter uv.

What is `uv`? The `npm` for Python You've Been Wishing For

uv is an extremely fast Python package installer and resolver, written in Rust by the same folks behind the ruff linter. Think of it as a single binary that aims to replace pip, pip-tools, and venv.

For a Node.js developer, the pitch is simple:

It's Fast: Like, esbuild or swc fast. It uses a global cache and parallel processing to make dependency installation significantly quicker.
It's an All-in-One Tool: It handles installing packages, running commands, and managing virtual environments. No more juggling different tools. It's like npm and npx rolled into one.
It's Familiar: The concepts map almost directly to what you already know from npm.

Step 1: Installing `uv`

First things first, you need Python installed. Once that's sorted, installing uv is a one-liner.

# macOS / Linux
curl -LsSf https://astral.sh/uv/install.sh | sh

# Windows
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"

This single binary is all you need.

Step 2: Creating a Virtual Environment (Your New `node_modules`)

In Node.js, dependencies are installed locally into a node_modules directory by default. This is great for project isolation. Python's answer to this is the "virtual environment," which is essentially a folder containing a copy of the Python interpreter and all the packages for a specific project.

With uv, creating one is trivial.

# Create a project and move into it
mkdir my-python-project
cd my-python-project

# Create a virtual environment named .venv
uv venv

This creates a .venv folder. Think of .venv as your new node_modules directory. It's where all your project-specific Python packages will live, keeping your global space clean.

Just like .gitignore includes node_modules, you should add .venv to your .gitignore file.

Step 3: Activating the Environment

This is one small step that's different from the Node.js workflow. To use the tools and packages inside your virtual environment, you need to "activate" it.

# macOS / Linux
source .venv/bin/activate

# Windows (Command Prompt)
.venv\Scripts\activate.bat

# Windows (PowerShell)
.venv\Scripts\Activate.ps1

Your terminal prompt will change to indicate that you're now "inside" the virtual environment. Now, any python or pip command will use the versions inside .venv.

Step 4: Installing Packages (The `npm install` Moment)

Ready for that familiar npm install feeling? Let's install a package. The requests library is the axios of the Python world, so we'll start there.

# This is like `npm install requests`
uv pip install requests

You'll see uv resolve and install the package with incredible speed. If you inspect the .venv directory, you'll find requests and its dependencies in there, just like you'd find packages in node_modules.

To install a dev dependency, you can add it to a specific requirements file, similar to devDependencies. For now, let's stick to the basics.

Step 5: Saving Dependencies (`package.json` and `requirements.txt`)

In Node.js, your dependencies are automatically saved to package.json. uv follows the classic Python pattern of using a requirements.txt file.

You can generate this file after installing your packages.

# This is like creating a snapshot of your dependencies
uv pip freeze > requirements.txt

Your requirements.txt will now look something like this:

# requirements.txt
certifi==2023.11.17
charset-normalizer==3.3.2
idna==3.6
requests==2.31.0
urllib3==2.1.0

This file is your package.json's dependencies list. Anyone else on your team can now install the exact same dependencies using this file.

# This is like running `npm install` after cloning a repo
uv pip install -r requirements.txt

Step 6: Running Tools with `uvx` (The `npx` Equivalent)

One of the best parts of the Node.js ecosystem is npx, which lets you run a package's binary without installing it globally.

uv has a direct equivalent: uvx.

Let's try it with cowsay, a classic command-line toy.

# This is like `npx cowsay "Hello from uv!"`
uvx cowsay "Hello from uv!"

uvx will temporarily install cowsay, run the command, and then clean it up. It's perfect for running project-based tools like linters (ruff), formatters (black), or web servers (uvicorn) without polluting your environment.

Tying It All Together: A Node.js to `uv` Cheat Sheet

Node.js (`npm`)	Python (`uv`)	Description
`npm install <pkg>`	`uv pip install <pkg>`	Installs a package to the local environment.
`npm install`	`uv pip install -r requirements.txt`	Installs all dependencies from the project file.
`package.json`	`requirements.txt`	The file that lists your project's dependencies.
`node_modules/`	`.venv/`	The local, isolated folder containing dependencies.
`npx <cmd>`	`uvx <cmd>`	Executes a package command without a permanent install.
`npm init -y`	`uv venv`	Initializes a new project environment.

Conclusion

The Python ecosystem is powerful, but for a Node.js developer, the tooling can sometimes feel like a barrier. uv changes that. By providing a single, cohesive, and lightning-fast tool, it offers a developer experience that feels right at home.

So next time you start a Python project, give uv a try. You might just find the npm-like workflow you've been looking for.

Explicit is Better Than Implicit: Mastering Pytest Fixtures and Async Testing

Chandrashekhar Kachawa — Wed, 03 Dec 2025 09:03:02 +0000

Look, we need to talk about async testing in Python. If you've ever stared at a ModuleNotFoundError: No module named 'trio' when you're just trying to test your FastAPI endpoints, you know the pain. You didn't ask for trio. You don't even know what trio is. You just want your tests to pass.

Let's fix this properly.

The Problem: Event Loops Are Confusing

Here's the thing about async Python: your async/await code needs an event loop to run. That's not optional. And there are multiple event loop implementations out there:

asyncio - Python's built-in standard library solution
trio - Third-party library with structured concurrency
curio - Another third-party option (less common these days)

When you're testing async code with pytest, you need something to manage these event loops. Enter pytest-anyio - a plugin that's supposed to make your life easier. Spoiler alert: it doesn't always.

What Actually Are Pytest Fixtures?

Before we dive into the async madness, let's talk about fixtures. If you've used pytest, you've probably seen them. If you haven't understood them, you're not alone.

Fixtures are just functions that set up stuff for your tests. That's it. They run before your test, give your test what it needs, and clean up after. Think of them as the responsible adult in the room.

Here's what makes them powerful:

Dependency Injection (The Good Kind)

You want a database connection? Just ask for it:

def test_something(database_connection):
    # pytest automatically gives you the connection
    result = database_connection.query("SELECT * FROM users")
    assert result is not None

No manual setup. No global variables. Just declare what you need, and pytest handles it.

Setup and Teardown Without the Boilerplate

Remember the old setUp() and tearDown() methods? Fixtures do this better:

@pytest.fixture
async def create_test_tables():
    # Setup: runs before your test
    async with engine.begin() as conn:
        await conn.run_sync(SQLModel.metadata.create_all)

    yield  # Your test runs here

    # Teardown: runs after your test
    async with engine.begin() as conn:
        await conn.run_sync(SQLModel.metadata.drop_all)

Everything before yield is setup. Everything after is cleanup. Your test gets a clean database every time.

Scopes: Because Not Everything Needs to Run Every Time

Fixtures have scopes that control how often they run:

function (default) - Runs for every single test
class - Runs once per test class
module - Runs once per test file
session - Runs once for your entire test suite

This matters for performance. Creating a database connection once per session? Smart. Creating it 500 times for 500 tests? Not smart.

The anyio Backend Problem

Now here's where things get spicy. When you use pytest-anyio to test async code, it needs to know which event loop to use. By default, it tries to be helpful and test against all available backends.

Sounds great, right? Wrong.

Here's what actually happens:

You write tests using asyncio (because that's what FastAPI uses)
You install pytest-anyio to test your async code
pytest-anyio sees you have trio installed (maybe from another project)
It tries to run your tests with trio
Your aiosqlite database driver explodes because it only works with asyncio
You get errors like ModuleNotFoundError: No module named 'trio' or RuntimeError: no running event loop
You question your career choices

The Solution: Be Explicit

Stop letting pytest-anyio guess. Tell it exactly what you want.

Step 1: Create an anyio_backend Fixture

In your tests/conftest.py, add this:

import pytest

@pytest.fixture(scope="session")
def anyio_backend():
    """
    Use asyncio for all async tests. Period.
    """
    return "asyncio"

That's it. This fixture runs once per test session and tells pytest-anyio: "Use asyncio. Don't try to be clever."

Step 2: Make Sure trio Isn't Interfering

If you don't need trio, uninstall it:

pip uninstall trio

Step 3: Mark Your Async Tests Properly

When you write async tests, mark them explicitly:

@pytest.mark.anyio(backend='asyncio')
async def test_create_and_get_example(client, create_test_tables):
    # Your test code here
    pass

The backend='asyncio' parameter is redundant if you have the fixture, but it makes your intent crystal clear.

Real-World Example: Testing FastAPI with SQLModel

Let's put this all together with a real example. You've got a FastAPI app with a database, and you want to test it properly.

The Database Setup Fixture

# tests/conftest.py
import pytest
from sqlmodel import SQLModel
from tests.test_db import engine

@pytest.fixture(scope="session")
def anyio_backend():
    return "asyncio"

@pytest.fixture
async def create_test_tables():
    """
    Creates tables before each test, drops them after.
    Each test gets a clean database.
    """
    async with engine.begin() as conn:
        await conn.run_sync(SQLModel.metadata.create_all)
    yield
    async with engine.begin() as conn:
        await conn.run_sync(SQLModel.metadata.drop_all)

The Test Client Fixture

# tests/features/test_examples_api.py
import pytest
from fastapi.testclient import TestClient
from src.app import app
from src.features.examples.api import get_example_service
from src.features.examples.service import ExampleService
from tests.test_db import TestingSessionLocal

async def override_get_example_service():
    """Override the dependency to use test database"""
    async with TestingSessionLocal() as session:
        yield ExampleService(session=session)

app.dependency_overrides[get_example_service] = override_get_example_service

@pytest.fixture(scope="module")
def client():
    with TestClient(app) as c:
        yield c

The Actual Test

@pytest.mark.anyio(backend='asyncio')
async def test_create_and_get_example(client: TestClient, create_test_tables):
    # Create an example
    create_response = client.post(
        "/api/v1/examples/",
        json={"name": "Test Example", "description": "A test description"},
    )
    assert create_response.status_code == 200
    created_example = create_response.json()
    assert created_example["name"] == "Test Example"
    assert "id" in created_example

    # Fetch it back
    example_id = created_example["id"]
    get_response = client.get(f"/api/v1/examples/{example_id}")
    assert get_response.status_code == 200
    fetched_example = get_response.json()
    assert fetched_example["id"] == example_id
    assert fetched_example["name"] == "Test Example"

Notice how the test just declares what it needs (client, create_test_tables), and pytest handles the rest. Clean database, test client, all set up and torn down automatically.

What We Fixed (The Troubleshooting Story)

When I first set this up, tests were failing with trio errors. Here's what was wrong and how we fixed it:

Problem 1: pytest-anyio was trying to use trio

Solution: Created the anyio_backend fixture returning "asyncio"

Problem 2: trio was installed but incompatible with aiosqlite

Solution: Uninstalled trio since we don't need it

Problem 3: Conflicting configurations in pyproject.toml and pytest.ini

Solution: Removed old anyio configurations, kept it simple

The lesson? Explicit is better than implicit. Don't let your test framework guess what you want.

Key Takeaways

Fixtures are dependency injection for tests - Declare what you need, pytest provides it
Use yield for setup/teardown - Code before yield is setup, code after is cleanup
Choose the right scope - Don't recreate expensive resources for every test
Be explicit about your async backend - Create an anyio_backend fixture that returns "asyncio"
Don't install backends you don't use - If you're using asyncio, you probably don't need trio
Mark async tests properly - Use @pytest.mark.anyio(backend='asyncio')

The Bottom Line

Async testing in Python doesn't have to be painful. The key is understanding what's actually happening:

Your async code needs an event loop
pytest-anyio manages that loop for you
You need to tell it which loop to use
Fixtures handle setup and teardown automatically

Get these pieces right, and your tests will be reliable, fast, and maintainable. Get them wrong, and you'll be debugging event loop errors at 2 AM.

Choose wisely.

Have you dealt with async testing nightmares? What tripped you up? Let me know in the comments or hit me up on Twitter. And if this saved you from a debugging session, you're welcome.

A Developer's Guide to Jest: What, When, and How to Test

Chandrashekhar Kachawa — Fri, 28 Nov 2025 07:56:52 +0000

Jest is a delightful JavaScript Testing Framework with a focus on simplicity. It's one of the most popular choices in the JavaScript ecosystem, especially within the React community, but its power and versatility make it a great fit for nearly any JavaScript project. This guide will walk you through how to use Jest, what to test with it, and when you might need to reach for a different tool.

Getting Started with Jest

Setting up Jest is incredibly straightforward. In a new or existing project, you can add it with your favorite package manager:

# Using npm
npm install --save-dev jest

# Using pnpm
pnpm add -D jest

# Using yarn
yarn add --dev jest

Next, add a test script to your package.json:

{
  "scripts": {
    "test": "jest"
  }
}

Now, let's write a simple test. Imagine you have a utility function that adds two numbers.

sum.js

function sum(a, b) {
  return a + b;
}
module.exports = sum;

To test this, you would create a file named sum.test.js. Jest automatically discovers and runs files with .test.js or .spec.js extensions.

sum.test.js

const sum = require('./sum');

describe('sum', () => {
  it('should add two numbers correctly', () => {
    expect(sum(1, 2)).toBe(3);
  });

  it('should handle negative numbers', () => {
    expect(sum(-1, -1)).toBe(-2);
  });
});

When you run pnpm test, Jest will execute this file and give you a clean, readable output indicating whether your tests passed or failed.

Core Concepts

describe(name, fn): Creates a block that groups together several related tests.
it(name, fn) or test(name, fn): This is your actual test case.
expect(value): The starting point for any assertion. When you wrap a value in expect, you get access to "matchers."
Matchers: Functions that let you validate a value in different ways. In the example above, .toBe(value) is a matcher that checks for exact equality. Other common matchers include toEqual (for deep object equality), toBeTruthy, toBeFalsy, toContain, and many more.

What to Test with Jest

Jest excels at unit testing and integration testing in a Node.js environment. Here are the ideal candidates for Jest tests:

Pure Functions & Business Logic: This is Jest's sweet spot. Functions that take inputs and produce outputs without side effects (like API calls or file system changes) are easy to test and form the backbone of a reliable application.
Component Logic: For UI frameworks like React, Vue, or Svelte, you can test the logic inside your components. For example, you can check if a component's state updates correctly when a function is called, without actually rendering the component to a screen.
API Endpoints: In a Node.js backend (like Express or Fastify), you can test your API endpoints by mocking requests and asserting that the correct status codes, headers, and JSON payloads are returned.

What Not to Test with Jest

It's equally important to know when Jest isn't the right tool for the job.

End-to-End (E2E) Testing: E2E testing involves simulating a real user's journey through your application in a browser. Since Jest runs in a Node.js environment (via JSDOM), it doesn't have a real browser engine. It cannot click buttons, fill out forms, and navigate pages as a user would.
- Use Instead: Tools like Cypress or Playwright are designed specifically for this. They run tests in actual browsers (Chrome, Firefox, etc.), providing much more reliable results for user-facing flows.
Visual Regression Testing: This is the practice of taking screenshots of your UI and comparing them against a baseline to catch unintended visual changes. While Jest's snapshot testing can capture the structure of a component's output, it knows nothing about its visual appearance.
- Use Instead: Tools like Percy or Chromatic integrate with E2E or component testing frameworks to perform robust visual testing.
Performance Testing: Jest is not built for benchmarking code performance. Its test runner overhead and focus on correctness make it unsuitable for measuring execution speed accurately.
- Use Instead: Use dedicated benchmarking libraries like Benchmark.js or profiling tools built into Node.js and browsers.

Advanced Features: Mocking and Snapshot Testing

Mocking: Jest has a powerful built-in mocking system (jest.fn(), jest.spyOn(), jest.mock()). Mocking allows you to replace dependencies (like an API service or a database module) with a "fake" version. This lets you test a piece of code in isolation, ensuring you're only testing its logic and not that of its dependencies.
Snapshot Testing: When you run a snapshot test, Jest generates a .snap file containing the rendered output of a component or the structure of an object. On subsequent test runs, Jest compares the new output to the saved snapshot. If they don't match, the test fails. This is useful for ensuring that UI component structures or large object shapes don't change unexpectedly. However, use it with caution—it's easy to accidentally approve incorrect snapshots.

Conclusion

Jest is a phenomenal tool for building a fast, reliable, and maintainable testing suite for your JavaScript codebase. It is the ideal choice for unit and integration tests, where you are verifying the logic and correctness of your functions, modules, and components.

However, a comprehensive testing strategy doesn't stop there. For verifying user journeys and visual presentation, you should complement Jest with specialized tools like Cypress, Playwright, or Percy. By using the right tool for the right job, you can build confidence in your code and ship better products.

Python Concurrency: A Guide to Threads, Processes, and Asyncio

Chandrashekhar Kachawa — Wed, 26 Nov 2025 05:35:47 +0000

Your Python script needs to do multiple things at once to be faster. But how? Python offers a rich but sometimes confusing landscape of concurrency and parallelism tools. Should you use threads, processes, or asyncio?

Choosing the right tool for the job is the key to writing efficient, scalable code. This guide will walk you through the three main concurrency models in Python, explaining what they are, how to use them, and when to choose each one.

The Foundation: `concurrent.futures`

For traditional, blocking code, Python's concurrent.futures module provides a beautiful, high-level API for managing pools of threads and processes. It introduces the "Executor" pattern, where you submit jobs to a pool and retrieve the results.

1. `ThreadPoolExecutor`: For I/O-Bound Work

When to use it: When your task spends most of its time waiting for external resources. This is called I/O-bound work. Examples include:

Making network requests (e.g., calling APIs, scraping websites).
Reading or writing from a slow disk.
Querying a database.

Due to Python's Global Interpreter Lock (GIL), threads are not suitable for speeding up CPU-heavy tasks, but they are perfect for I/O-bound tasks because they can "wait" concurrently.

Syntax Example: Let's download the content of several web pages.

import requests
from concurrent.futures import ThreadPoolExecutor

URLS = [
    "https://www.python.org/",
    "https://www.djangoproject.com/",
    "https://flask.palletsprojects.com/",
]

def fetch_url(url: str):
    print(f"Fetching {url}...")
    response = requests.get(url)
    print(f"Fetched {url} with status {response.status_code}")
    return len(response.content)

with ThreadPoolExecutor(max_workers=5) as executor:
    # The map function runs `fetch_url` for each item in URLS
    results = executor.map(fetch_url, URLS)

for url, length in zip(URLS, results):
    print(f"URL: {url}, Length: {length}")

The thread pool allows all three requests.get() calls to happen concurrently, dramatically speeding up the total execution time.

2. `ProcessPoolExecutor`: For CPU-Bound Work

When to use it: When your task is doing heavy computation and maxing out a CPU core. This is called CPU-bound work. Examples include:

Complex mathematical calculations.
Image or video processing.
Data analysis on large datasets.

Processes run in separate memory spaces and have their own Python interpreter, which allows them to bypass the GIL and run on different CPU cores in true parallel.

Syntax Example: Let's perform a heavy calculation on a list of numbers.

from concurrent.futures import ProcessPoolExecutor

def heavy_calculation(num: int):
    print(f"Calculating for {num}...")
    # A silly, CPU-intensive task
    return sum(i * i for i in range(num))

numbers_to_process = [100_000, 150_000, 200_000]

with ProcessPoolExecutor() as executor:
    results = executor.map(heavy_calculation, numbers_to_process)

for num, result in zip(numbers_to_process, results):
    print(f"Calculation for {num} returned {result}")

Notice the syntax is nearly identical to the thread pool example! concurrent.futures makes it easy to switch between them.

3. The Third Model: `asyncio`

What if your code is already async and uses libraries like httpx or asyncpg? In this case, you don't need threads or processes. asyncio has its own, more efficient tools for managing I/O-bound concurrency.

Managing Async Tasks: `gather` and `TaskGroup`

Let's say we have a simple async function:

async def say_after(delay: int, what: str):
    await asyncio.sleep(delay)
    print(what)

`asyncio.gather`

gather is a high-level utility to run multiple awaitable objects concurrently.

Gathering Coroutines (most common):
You can pass coroutine objects directly to gather. It will automatically schedule them as tasks.

import asyncio

# This runs both `say_after` calls concurrently
await asyncio.gather(
    say_after(1, "hello"),
    say_after(2, "world")
)

Gathering Tasks:
You can also create tasks explicitly with asyncio.create_task and then gather them. This gives you more control if you need to interact with the Task objects before they are complete.

task1 = asyncio.create_task(say_after(1, "hello"))
task2 = asyncio.create_task(say_after(2, "world"))
await asyncio.gather(task1, task2)

`asyncio.TaskGroup` (The Modern, Safe Way)

Introduced in Python 3.11, TaskGroup is a modern and safer way to manage concurrent tasks. It uses a context manager (async with) to create a scope, guaranteeing that all tasks created within it are awaited before the block is exited.

Benefits:

Structured Concurrency: No more "forgotten" tasks that run in the background.
Superior Exception Handling: If any task in the group fails, all other tasks are automatically cancelled.

Syntax Example:

async with asyncio.TaskGroup() as tg:
    tg.create_task(say_after(1, "hello"))
    tg.create_task(say_after(2, "world"))

print("Both tasks have now completed.")

For new asyncio code, TaskGroup is generally preferred over gather.

Summary: When to Use Which?

Is your task...	And are you using...	Your best tool is...
CPU-Bound	Any Python code	`ProcessPoolExecutor`
I/O-Bound	Blocking libraries (e.g., `requests`, `psycopg2`)	`ThreadPoolExecutor`
I/O-Bound	`async` libraries (e.g., `httpx`, `asyncpg`)	`asyncio` (`TaskGroup` or `gather`)

Conclusion

Python provides powerful tools for every concurrency and parallelism need. The key is to correctly identify the nature of your task. By choosing the right model—Processes for CPU-bound work, Threads for blocking I/O, and Asyncio for non-blocking I/O—you can write efficient, scalable, and high-performance applications.

Forem: Chandrashekhar Kachawa

System Design Fundamentals: From Monolith to Microservices

The Monolith: The All-in-One Application

Pros of a Monolith

Cons of a Monolith

Microservices: A Constellation of Services

Pros of Microservices

Cons of Microservices

The Migration: When and How?

The Strangler Fig Pattern

Conclusion

How to Design APIs

Core Architecture: A New Paradigm

1. Move Away from REST for Private APIs

2. Embrace a Single Endpoint and Batching

3. A Clean, Folder-Based Structure

The Request Flow: Security and Predictability First

1. Validation: The Most Critical Step

2. Smart Rate Limiting

3. Authorization: The First Line of Code

End-to-End Type Safety: The Holy Grail

Trade-Offs and Considerations

Pros

Cons

Conclusion

A Practical Guide to Backing Up Your Linux Server: PostgreSQL, Jenkins, and More

1. Backing Up Your PostgreSQL Database

On the Server: Creating the Backup

Downloading the Backup to Your Local PC

2. Backing Up Your Jenkins Server

3. Backing Up Essential System Files

4. Automating Your Backups with Cron

Conclusion

Mastering Real-Time Communication with Socket.IO Rooms

What are Rooms?

Setting Up the Basic Server

Room Creation and Uniqueness

How to Ensure Unique Rooms

Public vs. Private Rooms

Public Rooms

Private Rooms

When to Use Socket.IO Rooms

When NOT to Use Rooms

Conclusion

Hoppscotch: The Modern, Lightweight Alternative to Postman and Bruno

What is Hoppscotch?

Hoppscotch vs. Postman: A Battle of Philosophies

Hoppscotch vs. Bruno: A Different Kind of Open Source

Key Features That Make Hoppscotch Shine

Conclusion

Mastering Python Monorepos: A Practical Guide

Why a Monorepo?

Designing the Monorepo Structure

Creating a Shared Package

Integrating Your Services

1. FastAPI Microservice

2. Apache Airflow DAGs

Tooling and Best Practices with Ruff

Conclusion

Demystifying Retrieval-Augmented Generation (RAG)

What is RAG?

Why Do We Need It?

How Does RAG Work?

Step 1: Indexing (The Setup Phase)

Step 2: Retrieval and Generation (The Live Phase)

Conceptual Python Example

When to Use RAG

When Not to Use RAG

Life Update #1

From npm to uv: A Node.js Developer's Guide to Python Project Setup

What is uv? The npm for Python You've Been Wishing For

Step 1: Installing uv

Step 2: Creating a Virtual Environment (Your New node_modules)

Step 3: Activating the Environment

Step 4: Installing Packages (The npm install Moment)

Step 5: Saving Dependencies (package.json and requirements.txt)

Step 6: Running Tools with uvx (The npx Equivalent)

Tying It All Together: A Node.js to uv Cheat Sheet

Conclusion

Explicit is Better Than Implicit: Mastering Pytest Fixtures and Async Testing

What is `uv`? The `npm` for Python You've Been Wishing For

Step 1: Installing `uv`

Step 2: Creating a Virtual Environment (Your New `node_modules`)

Step 4: Installing Packages (The `npm install` Moment)

Step 5: Saving Dependencies (`package.json` and `requirements.txt`)

Step 6: Running Tools with `uvx` (The `npx` Equivalent)

Tying It All Together: A Node.js to `uv` Cheat Sheet

The Foundation: `concurrent.futures`

1. `ThreadPoolExecutor`: For I/O-Bound Work

2. `ProcessPoolExecutor`: For CPU-Bound Work

3. The Third Model: `asyncio`

Managing Async Tasks: `gather` and `TaskGroup`

`asyncio.gather`

`asyncio.TaskGroup` (The Modern, Safe Way)