Forem: Abhinav

Interesting read

Abhinav — Fri, 17 Apr 2026 06:42:07 +0000

Manthan Bhatt

Apr 12

What code review process do you use for AI-generated code?

#discuss #ai #productivity #development

Comments 15

1 min read

The Unseen Architects of Trust: Building Bulletproof Customer Experiences with Automated QA

Abhinav — Mon, 06 Apr 2026 08:15:55 +0000

We've all been there: a critical bug slips through, a feature goes live broken, and suddenly, our users are frustrated, our support channels are flooded, and our brand's reputation takes a hit. It's a feeling that can send shivers down any developer's spine. In the high-stakes world of customer-facing applications, where every interaction counts, ensuring quality isn't just good practice—it's paramount to building and maintaining user trust. But how do we guarantee this quality at scale, release after release, without sacrificing speed or innovation?

This is where automated Quality Assurance (QA) steps in, not merely as a safety net, but as the unseen architect of reliable, delightful customer experiences. It's about empowering our teams to move fast with confidence, knowing that our software consistently meets the high expectations our users have come to expect.

Why Automated QA is Non-Negotiable for Our Customer-Facing Apps

For applications that directly engage with users, the stakes are significantly higher. Manual testing simply can't keep pace with agile development cycles, diverse device landscapes, and the sheer volume of possible user interactions. Here's why we've made automated QA a cornerstone of our development philosophy:

Scale and Speed: As our applications grow and evolve, manual regression testing becomes a bottleneck. Automation allows us to run comprehensive test suites in minutes, not days, across various environments.
Consistency and Reliability: Automated tests execute the same steps every single time, eliminating human error and ensuring consistent validation of features. This builds confidence in our releases.
Early Bug Detection: By integrating tests into our CI/CD pipelines, we catch bugs much earlier in the development cycle, reducing the cost and effort of fixing them.
Confidence in Releases: When we have a robust automated test suite, we can deploy new features and bug fixes with significantly higher confidence, minimizing the risk of adverse customer impact.

Our Multi-Layered Approach to QA Automation

To truly build bulletproof applications, we adopt a multi-layered testing strategy, often visualized as the testing pyramid. Each layer serves a distinct purpose, contributing to the overall quality of our customer-facing apps.

1. Unit Tests: The Foundation of Trust

These are our fastest, most isolated tests, focusing on individual functions, methods, or components. They ensure that the smallest building blocks of our application behave exactly as intended. We primarily use frameworks like Jest (for JavaScript/TypeScript) or NUnit/xUnit (for .NET).

// Example: src/utils/cartCalculator.js
export function calculateTotalPrice(items) {
  if (!items || items.length === 0) {
    return 0;
  }
  return items.reduce((total, item) => total + (item.price * item.quantity), 0);
}

// Example: tests/unit/cartCalculator.test.js
import { calculateTotalPrice } from '../src/utils/cartCalculator';

describe('calculateTotalPrice', () => {
  test('should return 0 for an empty cart', () => {
    expect(calculateTotalPrice([])).toBe(0);
  });

  test('should correctly calculate total price for multiple items', () => {
    const items = [
      { price: 10, quantity: 2 },
      { price: 5, quantity: 3 }
    ];
    expect(calculateTotalPrice(items)).toBe(35); // (10*2) + (5*3) = 20 + 15 = 35
  });

  test('should handle single item correctly', () => {
    const items = [
      { price: 100, quantity: 1 }
    ];
    expect(calculateTotalPrice(items)).toBe(100);
  });
});

2. Integration Tests: Connecting the Dots

Integration tests verify that different modules or services in our application interact correctly with each other. This is crucial for customer-facing apps where front-end components often rely heavily on backend APIs. We use tools like Supertest (for Node.js APIs) or Postman/Newman for API collections.

// Example: tests/integration/api.test.js (using Supertest for an Express app)
import request from 'supertest';
import app from '../src/app'; // Our Express app instance

describe('Product API Integration Tests', () => {
  beforeAll(async () => {
    // Optionally, seed test database or start services
  });

  afterAll(async () => {
    // Optionally, clean up test database or stop services
  });

  test('GET /api/products should return a list of products', async () => {
    const res = await request(app).get('/api/products');
    expect(res.statusCode).toEqual(200);
    expect(res.body).toBeInstanceOf(Array);
    expect(res.body.length).toBeGreaterThan(0);
    expect(res.body[0]).toHaveProperty('id');
    expect(res.body[0]).toHaveProperty('name');
  });

  test('POST /api/products should create a new product', async () => {
    const newProduct = { name: 'Test Product', price: 29.99 };
    const res = await request(app)
      .post('/api/products')
      .send(newProduct);

    expect(res.statusCode).toEqual(201);
    expect(res.body).toHaveProperty('id');
    expect(res.body.name).toEqual('Test Product');
  });
});

3. End-to-End (E2E) Tests: User's Perspective

E2E tests simulate actual user journeys through our application, from logging in to completing a purchase. These are the most critical for customer-facing apps as they validate the entire system from the user's perspective. We rely heavily on robust tools like Playwright or Cypress for their speed, reliability, and excellent debugging capabilities.

// Example: tests/e2e/checkout.spec.js (using Playwright)
import { test, expect } from '@playwright/test';

test.describe('Checkout Flow', () => {
  test('should allow a user to complete a purchase', async ({ page }) => {
    await page.goto('http://localhost:3000/login');

    // Login
    await page.fill('input[name="email"]', 'testuser@example.com');
    await page.fill('input[name="password"]', 'password123');
    await page.click('button[type="submit"]');
    await expect(page).toHaveURL('http://localhost:3000/dashboard');

    // Add item to cart
    await page.click('text=Add to Cart'); // Assuming a button with this text
    await expect(page.locator('.cart-count')).toHaveText('1');

    // Navigate to cart and proceed to checkout
    await page.click('a[href="/cart"]');
    await page.click('button:has-text("Proceed to Checkout")');
    await expect(page).toHaveURL('http://localhost:3000/checkout');

    // Fill shipping details
    await page.fill('input[name="address"]', '123 Main St');
    await page.fill('input[name="city"]', 'Anytown');
    await page.selectOption('select[name="state"]', 'CA');
    await page.fill('input[name="zip"]', '90210');

    // Place order
    await page.click('button:has-text("Place Order")');

    // Verify order confirmation
    await expect(page).toHaveURL(/.*order-confirmation/);
    await expect(page.locator('h1')).toHaveText('Order Confirmed!');
  });
});

4. Performance Tests: Ensuring Responsiveness

Slow applications frustrate users and lead to abandonment. We integrate performance testing into our pipelines using tools like K6 or JMeter to simulate user load, identify bottlenecks, and ensure our application remains responsive even under peak traffic.

5. Accessibility Tests: Inclusive Experiences

Our customer-facing apps must be usable by everyone. We incorporate accessibility testing (e.g., using axe-core integrated with Playwright/Cypress) to ensure our applications meet WCAG standards, making them inclusive for users with disabilities.

Real-World Use Case: The Seamless Onboarding Flow

Consider our redesigned user onboarding process. Previously, a critical bug in the email verification step meant many new users couldn't complete registration, leading to high abandonment rates and frustrated support calls. Manual testing was proving insufficient to cover all edge cases and browser combinations.

By implementing a robust E2E test suite with Playwright, we created scenarios for successful registration, failed verification (e.g., expired token), re-sending verification emails, and different device orientations. These tests run automatically on every code push, catching regressions immediately. When a developer introduced a subtle change that broke the re-send email functionality, our E2E tests flagged it within minutes, long before it ever reached a staging environment. This allowed us to fix the issue proactively, ensuring a seamless onboarding experience for hundreds of new users daily.

Common Mistakes We've Learned to Avoid

While automation is powerful, it's not a silver bullet. We've stumbled and learned along the way:

Over-reliance on E2E Tests: While crucial, E2E tests are slow and brittle. Pushing too much logic into them makes our test suite slow and difficult to maintain. We prioritize the test pyramid, pushing as much testing as possible down to unit and integration layers.
Ignoring Test Data Management: Flaky tests often stem from inconsistent test data. We invest in strategies for generating, seeding, and cleaning up test data to ensure reliable test runs.
Lack of Test Maintenance: Automated tests are code and require maintenance. We allocate time for refactoring, updating, and removing obsolete tests, just like any other part of our codebase.
Poor Reporting and Feedback: Tests are only useful if their results are clear and actionable. We ensure our CI/CD pipelines provide immediate, easy-to-understand feedback on test failures, often with screenshots and video recordings for E2E tests.
Not Integrating into CI/CD: If tests aren't part of the automated build and deployment process, their value is severely diminished. Our tests run on every pull request and every push to our main branches, forming a mandatory gate for deployment.

TL;DR

Automated QA is essential for building trustworthy customer-facing apps. We use a multi-layered approach (unit, integration, E2E, performance, accessibility tests) to ensure quality at speed and scale. By simulating user journeys and catching bugs early, we deliver seamless experiences. Common pitfalls to avoid include over-reliance on E2E tests, poor test data management, and neglecting test maintenance; always integrate tests into CI/CD for maximum impact.

Conclusion: Our Commitment to Customer Trust

Automated QA isn't just about finding bugs; it's about embedding quality into the very fabric of our development process. It's an ongoing commitment to our users, ensuring that every interaction with our applications is smooth, reliable, and delightful. By investing in robust automation, we empower our engineering teams to build, iterate, and innovate faster, all while acting as the unseen architects of the trust our customers place in us. Let's continue to build with confidence, knowing our automated guardians are always on watch, protecting the experience we deliver.

From Zero to Autonomous: How We Built Our AI-Powered Dev.to Content Engine

Abhinav — Sun, 05 Apr 2026 13:34:45 +0000

From Zero to Autonomous: How We Built Our AI-Powered Dev.to Content Engine

Every developer knows the struggle: you've built something incredible, learned a ton, and now you want to share it with the world. But then reality hits. The blank page. The endless rephrasing. The looming deadline for your next blog post. Sound familiar? We've been there, staring at an empty editor, wishing our content could just... write itself. So, we decided to make that wish a reality by building a fully autonomous Dev.to publishing platform.

TL;DR

We built an autonomous Dev.to publishing platform! Our NestJS TypeScript backend uses Google Gemini to generate structured JSON for blog posts and email drafts. A sleek, glassmorphic Vanilla JS frontend provides a crucial human approval workflow before publishing directly to Dev.to via its REST API. This post details our architecture, the integration challenges we overcame, and the strategic choices behind our stack.

The Vision: Automating Content, Not Creativity

Our goal wasn't to replace human writers, but to empower them. We wanted a tool that could rapidly generate high-quality drafts, allowing our team to focus on refining ideas, adding unique perspectives, and ensuring factual accuracy, rather than wrestling with initial drafts and markdown formatting. This led us to envision a platform that handles the repetitive heavy lifting, providing a polished starting point for every piece of content.

Architectural Blueprint: A Symphony of Stacks

Building this system required a thoughtful combination of technologies, each chosen for its specific strengths and how it would contribute to a seamless, robust content pipeline.

Frontend: Vanilla JS & Glassmorphism - Why Go Barebones?

For our internal content management interface, we wanted absolute control over every pixel and interaction without the overhead of a large framework. This led us to choose Vanilla JavaScript.

Performance & Control: Pure JS offered unparalleled control and minimal bundle size, crucial for a snappy, responsive internal tool.
Custom UI/UX: We designed a custom, glassmorphic interface, giving our approval dashboard a premium, modern feel. The visual appeal of our internal tools was surprisingly important for team adoption.
Strict Approval Workflow: This was paramount. The frontend's primary role is to present generated content clearly, allow for easy edits, and facilitate a definitive 'Approve' or 'Reject' decision before anything goes live. It communicates directly with our NestJS backend for content generation and publishing actions.

Backend: NestJS & TypeScript - Our Robust Command Center

For the brains of our operation, we needed something robust, scalable, and developer-friendly. NestJS with TypeScript was a natural fit.

Modularity & Scalability: NestJS's modular architecture, inspired by Angular, allowed us to logically separate concerns into distinct modules (e.g., AiModule, DevtoModule, AuthModule). This structure makes the application easy to maintain, test, and scale.
Type Safety: TypeScript was crucial. When dealing with AI outputs that should conform to a specific JSON schema, type definitions provide compile-time checks and significantly reduce runtime errors.
Dependency Injection: NestJS's powerful DI system simplified component management and testing.
API Endpoints: Our backend exposes endpoints for initiating content generation (/ai/generate-post), managing drafts (/posts/:id/save-draft), and handling the final publication process (/posts/:id/publish).

AI Core: Google Gemini API - The Brain of Our Operation

This is where the magic happens. We chose Google Gemini for its multimodal capabilities and excellent fidelity in generating structured JSON output.

Structured Output: Our primary use case involved instructing Gemini to produce content strictly in JSON format, adhering to the Dev.to article schema (title, body_markdown, tags, summary). This was a game-changer for automating content processing.
Versatility: Beyond blog posts, we leveraged Gemini to draft email newsletters, social media captions, and even internal documentation, all tailored to specific formats and tones.
Prompt Engineering: The quality of the output is directly proportional to the quality of the prompt. We spent considerable time refining our prompts, explicitly defining the desired JSON schema and content requirements, including markdown formatting guidelines.

The Glue: Dev.to REST API Integration

Finally, connecting our generated and approved content to its destination: Dev.to. The Dev.to REST API is straightforward and well-documented.

Article Management: We utilized endpoints like POST /api/articles to create new posts (initially as drafts) and PUT /api/articles/:id to update them and finally mark them as published.
Secure Authentication: API keys were securely managed as environment variables on our NestJS backend, ensuring that our platform could interact with Dev.to on behalf of our team without exposing credentials client-side.

Deep Dive: Key Integrations & Code Snippets

Let's look at some simplified code snippets to illustrate how these components interact.

Gemini Integration Example (Simplified)

Our AiService handles the communication with the Google Gemini API, focusing on requesting structured JSON.

// backend/src/ai/ai.service.ts
import { Injectable } from '@nestjs/common';
import { GoogleGenerativeAI } from '@google/generative-ai';

@Injectable()
export class AiService {
  private readonly genAI: GoogleGenerativeAI;

  constructor() {
    this.genAI = new GoogleGenerativeAI(process.env.GEMINI_API_KEY);
  }

  async generateBlogPostContent(topic: string, keywords: string[]): Promise<any> {
    const model = this.genAI.getGenerativeModel({ model: "gemini-pro" });
    const prompt = `Generate a Dev.to blog post in JSON format based on the topic "${topic}" and keywords "${keywords.join(', ')}". The JSON should adhere to the following schema: { "title": "string", "body_markdown": "string", "tags": ["string"], "summary": "string" }. Ensure body_markdown uses clean markdown, including headings, code blocks, and lists. Max 4 tags, alphanumeric only.`;

    try {
      const result = await model.generateContent(prompt);
      const response = await result.response;
      const text = response.text();

      // Gemini often wraps JSON in markdown fences, so we need to extract it
      const jsonMatch = text.match(/```
{% endraw %}
json\n([\s\S]*?)\n
{% raw %}
```/);
      if (jsonMatch && jsonMatch[1]) {
          return JSON.parse(jsonMatch[1]);
      } else {
          // Fallback: try parsing directly if no markdown fences (less common but possible)
          return JSON.parse(text);
      }
    } catch (error) {
      console.error("Failed to parse JSON from Gemini output or API error:", error);
      throw new Error("AI content generation failed.");
    }
  }
}

Dev.to API Interaction (Simplified)

Our DevtoService orchestrates interaction with the Dev.to REST API.

// backend/src/devto/devto.service.ts
import { Injectable, HttpService } from '@nestjs/common';
import { firstValueFrom } from 'rxjs';

@Injectable()
export class DevtoService {
  constructor(private readonly httpService: HttpService) {}

  async createDevtoArticle(postData: any): Promise<any> {
    const headers = {
      'api-key': process.env.DEVTO_API_KEY,
      'Content-Type': 'application/json'
    };

    const payload = {
      article: {
        title: postData.title,
        body_markdown: postData.body_markdown,
        tags: postData.tags,
        published: false, // Always publish as draft first for approval
        canonical_url: postData.canonical_url, // Optional
        series: postData.series // Optional
      }
    };

    try {
      const response = await firstValueFrom(
        this.httpService.post('https://dev.to/api/articles', payload, { headers }),
      );
      return response.data;
    } catch (error) {
      console.error('Failed to create Dev.to article:', error.response?.data || error.message);
      throw new Error('Dev.to API error: Could not create article.');
    }
  }

  async publishDevtoArticle(articleId: number, articleData: any): Promise<any> {
    const headers = {
      'api-key': process.env.DEVTO_API_KEY,
      'Content-Type': 'application/json'
    };

    const payload = {
      article: {
        ...articleData, // Allow updating title, body, tags, etc. before publishing
        published: true // Mark as published
      }
    };

    try {
      const response = await firstValueFrom(
        this.httpService.put(`https://dev.to/api/articles/${articleId}`, payload, { headers }),
      );
      return response.data;
    } catch (error) {
      console.error('Failed to publish Dev.to article:', error.response?.data || error.message);
      throw new Error('Dev.to API error: Could not publish article.');
    }
  }
}

Approval Workflow Logic

The full workflow looks something like this:

Request Generation: A user in the frontend submits a topic and keywords to our NestJS backend (/ai/generate-post).
AI Generates Draft: The AiService calls Gemini, receives the JSON, and the backend stores this as a new Post entity in our database (e.g., MongoDB, PostgreSQL) with a status: 'pending_review'.
Review & Edit: The frontend fetches pending_review posts, displaying them in a markdown-rendered preview within our glassmorphic UI. Users can edit the title, body, tags, and summary directly.
Draft to Dev.to: Upon initial generation, the backend can also immediately create a private draft on Dev.to using devtoService.createDevtoArticle({ ...postData, published: false }), storing the article_id in our database.
Approval/Rejection: An authorized user (admin/editor) reviews the post. If approved, the (potentially edited) content is sent to our backend's publishPost endpoint, which then calls devtoService.publishDevtoArticle(articleId, updatedArticleData). If rejected, the internal status is simply updated to rejected, and the Dev.to draft is deleted or left untouched.

Challenges We Faced (And How We Overcame Them)

Building an autonomous system, even with a human in the loop, wasn't without its hurdles.

Prompt Engineering for Consistent JSON

Gemini is powerful, but it needs clear instructions. Initially, we sometimes received malformed JSON or content that didn't strictly adhere to our schema. We overcame this by:

Explicit Schema Definition: Including the exact JSON schema in the prompt.
Instruction Reinforcement: Adding phrases like "Always respond in JSON format" and "Do not include conversational text outside the JSON block."
Output Parsing: Implementing robust regex (/json\n([\s\S]*?)\n/) to reliably extract the JSON string, even when Gemini wrapped it in markdown code fences.

Rate Limiting & Error Handling

Both the Gemini and Dev.to APIs have rate limits. Hitting these could disrupt our workflow.

Retries with Exponential Backoff: We implemented a retry mechanism with exponential backoff for API calls, especially for the Gemini API.
Graceful Error Reporting: All API errors are caught, logged, and friendly error messages are passed back to the frontend, preventing a broken user experience.
Monitoring: We set up basic monitoring for API usage to anticipate and address potential rate limit issues before they become critical.

Security Considerations

Dealing with external APIs and content generation, security was paramount.

Environment Variables: All API keys (Gemini, Dev.to) are stored as environment variables and never committed to version control or exposed client-side.
Input Validation & Sanitization: Strict validation on all incoming data to our backend, preventing injection attacks and ensuring data integrity.
Authentication & Authorization: Our internal frontend required user authentication, and only authorized roles could perform actions like publishing to Dev.to.

UI/UX for Content Review

Making the approval process intuitive was vital. If reviewing was cumbersome, our team wouldn't use it.

Real-time Markdown Preview: As users edited the body_markdown, a live preview showed exactly how it would look on Dev.to.
Editable Fields: Every part of the generated JSON (title, body, tags, summary) was easily editable.
Clear Status Indicators: Visual cues for pending_review, approved, rejected, published states.

Why This Stack? A Strategic Choice

Our technology choices were deliberate, aimed at maximizing efficiency, scalability, and developer experience:

NestJS/TypeScript: Provided the robustness, scalability, and type safety needed for a critical backend service. Its enterprise-grade features and strong community made it a confident choice for handling complex logic and external API integrations.
Google Gemini: Positioned as a leading AI model, Gemini offered the necessary capabilities for structured content generation with high accuracy. Its continuous improvement aligns with our long-term vision for the platform.
Vanilla JS Frontend: For a highly focused internal tool, eschewing a heavy framework meant less boilerplate, faster development, and absolute control over performance and aesthetics. It proved to be a lightweight yet powerful choice for building our custom glassmorphic UI.

Real-world Use Case: Beyond Blog Posts

While this platform was initially designed for Dev.to posts, the core architecture is highly adaptable. We've already started exploring its potential for:

Internal Documentation: Generating initial drafts for technical specifications, onboarding guides, or API documentation.
Marketing Copy: Quickly producing variations of ad headlines, social media posts, or website copy.
Email Newsletters: Drafting segment-specific email content, saving considerable time for our marketing team.
Code Snippet Generation: For tutorials or quick examples, AI can provide a useful starting point.

Common Mistakes to Avoid

If you're considering building something similar, learn from our experience:

Over-relying on Raw AI Output: Always, always have a human review step. AI is a co-pilot, not a replacement. Unvetted AI content can lead to misinformation or brand damage.
Ignoring Approval Workflows: A

Unleashing the Content Kraken: Building an Autonomous Dev.to Publishing Engine with NestJS & Gemini

Abhinav — Sun, 05 Apr 2026 13:31:25 +0000

Unleashing the Content Kraken: Building an Autonomous Dev.to Publishing Engine with NestJS & Gemini

Ever stared at a blank editor, the cursor blinking accusingly, knowing you should be publishing engaging technical content but feeling the overwhelming weight of research, writing, and formatting? We've been there. The desire to share knowledge is strong, but the consistent effort required to produce high-quality, dev.to-ready posts can often feel like a full-time job in itself. What if there was a way to significantly streamline that process, injecting AI assistance while maintaining human oversight?

That's the exact problem we set out to solve. Our ambition: to build a fully autonomous, yet strictly controlled, Dev.to publishing platform from scratch. We envisioned a system that could dynamically generate structured blog posts and even email drafts, all wrapped in a custom, intuitive interface. This post details our journey, the architectural choices, the integration hurdles, and why our specific stack – NestJS, Google Gemini API, and Vanilla JS – proved to be the perfect combination.

The Grand Vision: An End-to-End Content Machine

Our goal wasn't just to write posts with AI; it was to create a complete content lifecycle management system. From ideation (via AI prompts) to generation, review, approval, and finally, publication on Dev.to, we wanted a seamless flow. The core pillars of our vision included:

AI-Powered Content Generation: Leverage large language models to draft engaging technical articles and accompanying promotional emails.
Structured Output: Ensure the AI generated content adhered to a specific JSON schema, making it easily consumable by our platform.
Strict Human Approval: A critical

Debugging Node.js Out-of-Memory Crashes: A Practical, Step-by-Step Story

Abhinav — Tue, 30 Dec 2025 00:13:02 +0000

How we tracked down a subtle memory leak that kept taking our production servers down—and how we fixed it for good.

The OOM That Ruined a Monday Morning

Everything looked normal—until alerts started firing. One by one, our Node.js API instances were crashing with a familiar but dreaded message:

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

The pattern was frustratingly consistent:

Servers ran fine for hours
Traffic increased
Memory climbed steadily
Then 💥—a crash

If you’ve ever dealt with Node.js in production, you already know what this smells like: a memory leak.

In this post, I’ll walk through exactly how we diagnosed the problem, what signals mattered most, and the simple fix that stabilized memory under heavy load.

Reading the GC Tea Leaves

Before touching any code, we looked closely at the garbage collector output from V8:

Mark-Compact (reduce) 646.7 (648.5) -> 646.6 (648.2) MB

At first glance, it looks harmless. But the key insight was this:

GC freed almost nothing.

From ~646.7 MB to ~646.6 MB. That’s essentially zero.

What that tells us

GC is running frequently and expensively
Objects are still strongly referenced
Memory is not eligible for collection

In short: this is not “GC being slow”—this is leaked or over-allocated memory.

Preparing the Battlefield

1. Confirm the Heap Limit

First, we verified how much memory Node.js was actually allowed to use:

const v8 = require('v8');

console.log(
  'Heap limit:',
  Math.round(v8.getHeapStatistics().heap_size_limit / 1024 / 1024),
  'MB'
);

This removes guesswork—especially important in containers or cloud runtimes.

2. Turn on GC Tracing

Next, we watched GC behavior in real time:

node --trace-gc server.js

This shows:

Scavenge → minor GC (young objects)
Mark-Sweep / Mark-Compact → major GC (old generation)

Frequent major GCs with poor cleanup are a huge red flag.

3. Shrink the Heap (On Purpose)

Instead of waiting hours for production crashes, we forced the issue locally:

node --max-old-space-size=128 server.js

A smaller heap means memory problems surface fast—often in minutes.

4. Reproduce with Load

We wrote a simple concurrent load script to mimic real traffic. Under load, memory climbed steadily and never came back down.

At this point, we had a reliable reproduction. Time to hunt the leak.

Template: Load Test Script for Memory Testing

To consistently reproduce the issue locally (instead of waiting for real traffic), we used the following load test template.

This script is intentionally minimal:

No external dependencies
Configurable concurrency
Responses are fully consumed (important for memory accuracy)
Designed for GC and heap behavior, not benchmarking

Usage

node load-test.js [concurrent] [endpoint]

# Example
node load-test.js 100 data

Load Test Template Code

RES:

/**
 * Load Test Script for Memory Testing
 * Usage: node load-test.js [concurrent] [endpoint]
 * Example: node load-test.js 100 data
 */

const http = require('http');

const CONFIG = {
  hostname: 'localhost',
  port: 3000,
  endpoints: {
    data: {
      path: '/api/data',
      method: 'POST',
      headers: {
        'Content-Type': 'application/json'
      },
      body: JSON.stringify({
        items: ['sample_item'],
        userContext: {
          userId: 'test-user',
          sessionId: 'test-session'
        }
      })
    }
  }
};

const CONCURRENT = parseInt(process.argv[2]) || 50;
const ENDPOINT = process.argv[3] || 'data';

const endpointConfig = CONFIG.endpoints[ENDPOINT];
if (!endpointConfig) {
  console.error(
    `Unknown endpoint: ${ENDPOINT}. Available: ${Object.keys(CONFIG.endpoints).join(', ')}`
  );
  process.exit(1);
}

const makeRequest = (requestId) => {
  return new Promise((resolve) => {
    const startTime = Date.now();

    const options = {
      hostname: CONFIG.hostname,
      port: CONFIG.port,
      path: endpointConfig.path,
      method: endpointConfig.method,
      headers: endpointConfig.headers
    };

    const req = http.request(options, (res) => {
      // Consume response to avoid socket & memory leaks
      res.resume();

      res.on('end', () => {
        resolve({
          requestId,
          status: res.statusCode,
          duration: Date.now() - startTime,
          success: res.statusCode >= 200 && res.statusCode < 300
        });
      });
    });

    req.on('error', (err) => {
      resolve({
        requestId,
        success: false,
        duration: Date.now() - startTime,
        error: err.message
      });
    });

    req.setTimeout(30000, () => {
      req.destroy();
      resolve({
        requestId,
        success: false,
        duration: Date.now() - startTime,
        error: 'Timeout'
      });
    });

    if (endpointConfig.body) {
      req.write(endpointConfig.body);
    }

    req.end();
  });
};

const runLoadTest = async () => {
  console.log('='.repeat(60));
  console.log('MEMORY LOAD TEST');
  console.log('='.repeat(60));
  console.log(`Endpoint: ${endpointConfig.method} ${endpointConfig.path}`);
  console.log(`Concurrent Requests: ${CONCURRENT}`);
  console.log(`Target: ${CONFIG.hostname}:${CONFIG.port}`);
  console.log('='.repeat(60));
  console.log('\nStarting load test...\n');

  const startTime = Date.now();

  const promises = Array.from(
    { length: CONCURRENT },
    (_, i) => makeRequest(i + 1)
  );

  const results = await Promise.all(promises);
  const totalTime = Date.now() - startTime;

  const successful = results.filter(r => r.success);
  const failed = results.filter(r => !r.success);
  const durations = successful.map(r => r.duration);

  const avgDuration = durations.length
    ? Math.round(durations.reduce((a, b) => a + b, 0) / durations.length)
    : 0;

  console.log('='.repeat(60));
  console.log('RESULTS');
  console.log('='.repeat(60));
  console.log(`Total Requests:    ${CONCURRENT}`);
  console.log(`Successful:        ${successful.length}`);
  console.log(`Failed:            ${failed.length}`);
  console.log(`Total Time:        ${totalTime}ms`);
  console.log(`Avg Response Time: ${avgDuration}ms`);
  console.log(`Min Response Time: ${Math.min(...durations)}ms`);
  console.log(`Max Response Time: ${Math.max(...durations)}ms`);
  console.log(`Requests/sec:      ${Math.round(CONCURRENT / (totalTime / 1000))}`);
  console.log('='.repeat(60));

  if (failed.length) {
    console.log('\nFailed requests:');
    failed.slice(0, 5).forEach(r => {
      console.log(`  Request #${r.requestId}: ${r.error}`);
    });
  }

  console.log('\n>>> Check server logs for [MEM] entries <<<\n');
};

runLoadTest().catch(console.error);

Following the Memory Trail

We added lightweight logging around suspicious paths:

const logMemory = (label) => {
  const { heapUsed } = process.memoryUsage();
  console.log(`[MEM] ${label}: ${Math.round(heapUsed / 1024 / 1024)} MB`);
};

Under load, the logs told a clear story:

processData START: 85 MB
processData START: 92 MB
processData START: 99 MB
processData START: 107 MB

Memory kept climbing—request after request.

Eventually, all roads led to one innocent-looking helper function.

The Real Culprit

const getItemAssets = (itemType) => {
  const assetConfig = {
    item_a: { thumbnail: '...', full: '...' },
    item_b: { thumbnail: '...', full: '...' },
    // many more entries
  };

  return assetConfig[itemType] || { thumbnail: '', full: '' };
};

Why this was disastrous

The config object was recreated on every call
The function ran multiple times per request
Under concurrency, tens of thousands of objects were created per second

Even though GC could collect them, allocation happened faster than cleanup—pushing objects into the old generation and eventually exhausting the heap.

The Fix: One Small Move, Huge Impact

const ASSET_CONFIG = Object.freeze({
  item_a: { thumbnail: '...', full: '...' },
  item_b: { thumbnail: '...', full: '...' },
});

const DEFAULT_ASSET = Object.freeze({ thumbnail: '', full: '' });

const getItemAssets = (itemType) =>
  ASSET_CONFIG[itemType] || DEFAULT_ASSET;

What changed?

Objects are created once, not per request
Zero new allocations in the hot path
Dramatically reduced GC pressure

Proving the Fix Worked

We reran the exact same test:

Before

Heap climbed relentlessly
GC freed almost nothing
Process crashed at ~128 MB

After

Heap usage oscillated within a tight range
Minor GCs cleaned memory efficiently
No crashes—even under sustained load

Final Thought

Most Node.js OOM crashes aren’t caused by “huge data” or “bad GC.”
They’re caused by small, repeated allocations in the wrong place.

Once you learn to read GC logs and control allocation rate, memory bugs stop being mysterious—and start being fixable.

🔗 Designing a URL Shortener: From Architecture to Node.js Implementation

Abhinav — Sun, 14 Dec 2025 16:06:55 +0000

After practicing system design concepts for a while, I finally decided to apply them by building real projects. This URL shortener is my first assignment, where I focused on translating high-level design ideas into a working system—covering architecture, caching, databases, and scalability considerations. The goal was not just to build a feature, but to think through the trade-offs and design decisions that matter in production systems.

A URL shortener is one of those systems that looks trivial at first glance but reveals deep engineering challenges once we design it for scale, reliability, and performance.

In this blog, we’ll design a URL shortener end-to-end:

What a URL shortener is and why we need it
High-Level Design (HLD)
Database and caching strategy
Browser cache & HTTP redirects
Back-of-the-envelope calculations
A clean Node.js implementation with code

1️⃣ What Is a URL Shortener?

A URL shortener converts a long URL into a compact alias.

Example

Long URL:
https://www.example.com/products/electronics/mobiles/samsung/galaxy-s23-ultra

Short URL:
https://sho.rt/aZ9kQ

When someone opens the short URL, the system:

Finds the original long URL
Redirects the user to it

At its core, this is a key–value mapping:

short_code → long_url

2️⃣ Why Do We Need URL Shorteners?

🔹 Better User Experience

Long URLs are hard to read, remember, and share.

🔹 Platform Constraints

SMS, social media posts, QR codes, and print media benefit from shorter links.

🔹 Analytics

We can track:

Click counts
Time
Location
Device / browser

🔹 Link Management

We gain the ability to:

Expire links
Disable links
Rotate destinations
Run A/B tests

3️⃣ High-Level Design (HLD)

🧱 Logical Architecture

🔹 Why Each Component Exists

Browser Cache → Fastest cache (used carefully)
CDN → Global low latency
Load Balancer → Traffic distribution & failover
Service Layer → Stateless business logic
Redis → Ultra-fast lookups for hot links
Database → Durable storage

4️⃣ Database Design

📦 Data Model

Field	Description
short_code	Primary key
long_url	Original URL
created_at	Creation time
expiry_at	Optional
click_count	Optional

🗄️ SQL vs NoSQL

SQL → Simpler, strong consistency
NoSQL (DynamoDB / Cassandra) → Horizontal scale, high throughput

👉 At scale, this is a NoSQL key-value workload:

Partition Key = short_code

5️⃣ Server-Side Cache (Redis)

Redirect traffic is ~99% reads.

🔥 Why Redis?

Memory-level speed
Offloads database
Handles hot links efficiently

⚡ Redirect Flow

Request
  ↓
Redis hit → Redirect
Redis miss → DB lookup → Cache → Redirect

6️⃣ Browser Cache (Important Trade-off)

Browser cache sits before our servers and can completely bypass them.

Redirect Status Codes Matter

Status	Browser Behavior
301	Cached aggressively
302 / 307	Revalidated / not cached

Typical Production Strategy

We usually disable browser caching:

Cache-Control: no-store, no-cache, must-revalidate

Why?

We don’t lose analytics
We can revoke links
We can rotate destinations

👉 Speed comes from Redis + CDN, not browser cache.

7️⃣ Short Code Generation

Base62 Encoding

Characters:

a–z A–Z 0–9  (62 chars)

Benefits:

URL-safe
Compact
Deterministic

Example:

ID: 125 → cb

8️⃣ Back-of-the-Envelope Calculations

Assumptions

DAU: 10 million
URLs created/user/day: 0.1
Redirects per URL/day: 50

Write Traffic

1M URLs/day ≈ 12 writes/sec

Read Traffic

50M redirects/day ≈ 580 RPS
Peak ≈ 3,000 RPS

👉 System is read-heavy, so caching dominates design.

Storage

~260 bytes / URL
365M URLs/year ≈ 95 GB/year

Manageable with NoSQL.

Redis Cache

10M hot URLs × 300 bytes ≈ 3 GB

9️⃣ Node.js Implementation

📁 Project Structure

src/
 ├── app.ts
 ├── controllers/
 │    └── url.controller.ts
 ├── services/
 │    └── url.service.ts
 ├── repositories/
 │    └── url.repository.ts
 ├── cache/
 │    └── redis.client.ts
 └── utils/
      └── base62.ts

🔹 Base62 Encoder

// utils/base62.ts
const chars = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";

export function encodeBase62(num: number): string {
  let result = "";
  while (num > 0) {
    result = chars[num % 62] + result;
    num = Math.floor(num / 62);
  }
  return result;
}

🔹 Redis Client

// cache/redis.client.ts
import Redis from "ioredis";

export const redis = new Redis({
  host: "localhost",
  port: 6379
});

🔹 Repository Layer (DB abstraction)

// repositories/url.repository.ts
const db = new Map<string, string>(); // mock DB

export class UrlRepository {
  async save(shortCode: string, longUrl: string) {
    db.set(shortCode, longUrl);
  }

  async findByShortCode(code: string): Promise<string | null> {
    return db.get(code) || null;
  }
}

🔹 Service Layer (Core Logic)

// services/url.service.ts
import { UrlRepository } from "../repositories/url.repository";
import { redis } from "../cache/redis.client";
import { encodeBase62 } from "../utils/base62";

let globalId = 100000;

export class UrlService {
  private repo = new UrlRepository();

  async createShortUrl(longUrl: string): Promise<string> {
    const id = globalId++;
    const shortCode = encodeBase62(id);

    await this.repo.save(shortCode, longUrl);
    await redis.set(shortCode, longUrl);

    return shortCode;
  }

  async getLongUrl(shortCode: string): Promise<string | null> {
    const cached = await redis.get(shortCode);
    if (cached) return cached;

    const longUrl = await this.repo.findByShortCode(shortCode);
    if (!longUrl) return null;

    await redis.set(shortCode, longUrl);
    return longUrl;
  }
}

🔹 Controller Layer

// controllers/url.controller.ts
import { Request, Response } from "express";
import { UrlService } from "../services/url.service";

const service = new UrlService();

export async function shortenUrl(req: Request, res: Response) {
  const { longUrl } = req.body;
  const code = await service.createShortUrl(longUrl);

  res.json({
    shortUrl: `https://sho.rt/${code}`
  });
}

export async function redirect(req: Request, res: Response) {
  const { code } = req.params;
  const longUrl = await service.getLongUrl(code);

  if (!longUrl) {
    return res.status(404).send("Not found");
  }

  res.setHeader("Cache-Control", "no-store");
  res.redirect(302, longUrl);
}

🔹 App Entry Point

// app.ts
import express from "express";
import { shortenUrl, redirect } from "./controllers/url.controller";

const app = express();
app.use(express.json());

app.post("/shorten", shortenUrl);
app.get("/:code", redirect);

app.listen(3000, () => {
  console.log("URL Shortener running on port 3000");
});

1.POST /shorten

2.GET /code

🔟 Final Thoughts

A URL shortener is a perfect system design problem because it tests:

Data modeling
Caching strategy (browser, CDN, Redis)
HTTP semantics
Scale estimation
Clean backend architecture

The biggest lesson:

The fastest systems are not the ones with the most caches —
but the ones that cache at the right layers.

🧠 "The Pool is Full": A Deep Dive on a Knex.js Bug That Wasn't a Database Problem

Abhinav — Wed, 12 Nov 2025 05:52:01 +0000

It started, as these things always do, with a mysterious error message popping up in our production logs during peak hours:

KnexTimeoutError: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?

This is the kind of error that makes your stomach drop. It’s sporadic—sometimes during a traffic burst, sometimes when a background job kicks off. It's the worst kind of bug.

Our first instinct? "The database is overloaded. We're hitting our limits."

But as we dug in, we realized this wasn’t a database saturation problem at all. It was a subtle, tricky concurrency problem we had created in our own code.

🔍 The Setup (and Our False Sense of Security)

Here's a quick look at our stack for this service:

Knex.js as our query builder for PostgreSQL.
Mongoose for our MongoDB data.
A set of APIs that, for a single request, often need to fetch data from both databases.

To handle this, we had a pretty beefy Knex connection pool: 100 connections. On paper, this should have been more than enough for our workload.

And yet, here we were, getting timeouts as if the pool was completely exhausted.

🧩 Step 1: Finding the Pattern (The "Huh, That's Weird" Moment)

The timeouts weren't happening everywhere. They were clustered around a few specific endpoints. And the one thing they all had in common was this pattern:

// A simplified version of the "fast" code
await Promise.all([
  fetchFromPostgres(),  // Knex query
  fetchFromMongo()      // Mongoose query
]);

Looks smart, right? We're running our database calls in parallel to be efficient and reduce response time. That’s just good async programming!

...or so we thought. This assumption, it turns out, was the root of all our problems.

🧠 Step 2: How Knex Pools Actually Work

To understand why this broke, we had to go back to basics. How does Knex (which uses tarn.js under the hood) manage its pool?

Let's say our config is:

pool: {
  min: 2,
  max: 100,
  acquireTimeoutMillis: 60000 // 60 seconds
}

This means:

Knex will keep at most 100 connections open to Postgres.
When a query needs to run, it "checks out" a connection.
When the query is done, it "releases" the connection back to the pool.
If all 100 connections are checked out, the next query has to wait.
If it waits for more than 60 seconds... boom. KnexTimeoutError.

This is straightforward. But the chaos starts when you mix this simple checkout/release system with Node.js's event loop and other async tasks.

⚠️ Step 3: The "Aha!" Moment — The `Promise.all()` Trap

Here’s where it all fell apart.

When you await Promise.all(), you're telling Node.js: "Start both of these things right now."

So, for a single incoming API request, this happens:

fetchFromPostgres() kicks off. It goes to the Knex pool and acquires Connection #1.
fetchFromMongo() kicks off at the same time. It's a totally separate I/O task, hitting a different database.

Now, multiply this by 100 concurrent API requests all hitting this endpoint at once.

Request 1: Acquires Postgres Connection #1.
Request 2: Acquires Postgres Connection #2.
...
Request 100: Acquires Postgres Connection #100.
Request 101: Tries to get a connection. The pool is full. It starts waiting.

But here’s the real kicker: The fetchFromMongo() calls.

Those Mongoose queries are also taking up time and resources on the Node.js event loop. While Node is busy handling Mongo I/O, parsing large JSON responses, or doing anything else, it's delaying the very moment that the fetchFromPostgres() query finishes and releases its connection.

We weren't just running out of connections. We were holding onto them for way too long because the event loop was busy with other "parallel" work. Our "fast" code was causing a traffic jam.

🧮 Step 4: Why "Just Add More Connections" Would Have Failed

You might be thinking, "Why not just increase the pool to 500?"

We thought about it. But that would have just masked the symptom and likely caused a new, worse problem: saturating the PostgreSQL server itself.

Every connection in the Knex pool is a real backend process on your Postgres server, consuming memory. Hundreds of concurrent, active connections is not something Postgres handles efficiently. We would have just moved the bottleneck from our Node app to the database, and probably taken down the whole thing.

This was not a pool-size problem. It was a connection lifecycle problem.

⚙️ Step 5: The Fix (That Felt Too Simple)

Once we truly understood the problem, the fix was almost embarrassingly simple.

We stopped trying to be clever. We made the code sequential.

// ✅ The "boring" but stable version
const postgresData = await fetchFromPostgres();  // 1. Acquire, 2. Use, 3. RELEASE
const mongoData = await fetchFromMongo();        // 4. Now do other stuff

And just like that, the timeouts vanished.

This code looks slower, but it performs infinitely better under load. Why?

The Postgres connection is acquired and released in a tiny, predictable window.
The connection pool never gets overwhelmed. Most of the time, it's sitting at 20-30 active connections, not slammed at 100.
The Mongo query (the "other async work") now runs after the scarce resource (the Postgres connection) has been returned, so it can't cause a delay.

We traded theoretical, single-request parallelism for actual, system-wide throughput.

🧾 Step 6: The Proof

After deploying the fix, we watched the dashboards.

Result: Zero KnexTimeoutErrors in the two weeks since.
Connection Pool Metrics: Stable. Average active connections dropped from spiking at 100 to a healthy 20-30.
Bonus: Average response time during high-load actually dropped by ~15%. Why? Because requests weren't sitting around waiting 60 seconds for a connection to free up.

💡 My Hard-Won Lessons

This bug was a humbling reminder that in systems programming, "fast" is almost always more complicated than it looks.

"Parallel" Isn't Always Faster. Especially when you're all fighting for the same limited resource. It's like opening all the security lines at the airport but only having one TSA agent. The coordination is the bottleneck.
Node.js Has a Busy To-Do List. Node.js is single-threaded. Any long-running I/O or a heavy CPU task (like parsing a huge JSON from Mongo) can and will delay other work, like the callbacks that free your database connections.
You Can't Fix What You Can't See. Monitor your connection pool! We now have alerts on "active connections" and "pending acquires." If we didn't have those metrics, we'd still be guessing.
Treat Your Connection Pool Like a Scarce Resource. Keep your transactions and queries as short as humanly possible. Get in, get your data, and get out. Don't hold a connection open while you go off and do something else.

This whole ordeal started as a scary production fire, but it ended up being a fantastic lesson in how our async code really behaves under pressure. The way we coordinate our work is often far more important than the speed of any single part.

⚡ Scaling User Search with Bloom Filters in Node.js

Abhinav — Wed, 17 Sep 2025 09:06:55 +0000

When your system grows to millions of users, even the simplest operations—like checking if a phone number or email already exists—can become costly.

Yes, you can add database indexes, but every lookup still eats up I/O and CPU cycles. Under heavy signup traffic, this can quickly overwhelm your database.

This is where Bloom Filters come to the rescue. 🌸

🌱 What is a Bloom Filter?

A Bloom Filter is a probabilistic data structure used for set membership checks. It allows us to ask:

👉 “Does this value possibly exist?”

It can say:

❌ Definitely Not → Safe to skip DB.
✅ Might Exist → Confirm with DB.

This tiny compromise (allowing false positives, but never false negatives) gives us O(1) lookups with very little memory usage.

🔬 Anatomy of a Bloom Filter (Abscissa)

At its core, a Bloom filter is just:

A Bit Array (size m) → starts with all 0’s.
k Hash Functions → each maps an input to one of the m positions.

👉 When we add an element:

Run it through all k hash functions.
Flip those positions in the bit array to 1.

👉 When we check an element:

Run it through the same k functions.
If all those positions are 1 → the element might exist.
If any position is 0 → it definitely does not exist.

📈 Visual Abscissa

Think of it as a number line (abscissa = x-axis):

Bit Array (size m)
0 1 2 3 4 5 6 7 8 9 ... m-1
[0][0][0][0][0][0][0][0][0][0]

Each hash function picks some positions along this line.
Adding "alice@example.com" might flip positions 3, 7, and 9.
Checking "bob@example.com"? If one of its hash positions is still 0, we know Bob isn’t in the set.

⚖️ Balancing Act

More bits (m) → fewer collisions, lower false positives.
More hash functions (k) → more accuracy, but also more computation.
The sweet spot depends on expected number of elements n.

Formula for optimal k:

k=n/m(ln2)

This balance is why Bloom filters are tiny in memory yet mighty in scale.

🏗️ Our Architecture

We built a Bloom Filter Service in Node.js that acts as a fast gatekeeper before the database.

It consists of:

Routes Layer → API endpoints for clients.
Handler Layer → Processes requests, interacts with the service.
Service Layer → Manages Bloom filters, population, refresh, and lookups.

📜 Routes Layer

We expose three endpoints under /bloom_filter:

import express from 'express';
import { getBloomFilterStatus, refreshBloomFilter, checkPhoneExists } from './handler.js';

const router = express.Router();

router.get('/status', getBloomFilterStatus);
router.post('/refresh', refreshBloomFilter);
router.get('/check', checkPhoneExists);

export default router;

GET /status → Monitor filters.
POST /refresh → Force a refresh.
GET /check?phoneNumber=... → Check existence.

⚙️ Handler Layer

The handlers sit between API requests and the service. They manage errors and responses:

import userSearchBloomFilter from '../../services/userSearchBloomFilter.js';
import { generateInternalServerErrorRepsonse } from '../../utils/errorHandler.js';

export async function getBloomFilterStatus(req, res) {
    try {
        const status = userSearchBloomFilter.getStatus();
        return res.status(200).json({ success: true, data: status });
    } catch (error) {
        const errorResponse = await generateInternalServerErrorRepsonse(error, 'getBloomFilterStatus');
        return res.status(500).json(errorResponse);
    }
}

export async function refreshBloomFilter(req, res) {
    try {
        await userSearchBloomFilter.refresh();
        return res.status(200).json({
            success: true,
            message: 'Bloom filter refreshed successfully'
        });
    } catch (error) {
        const errorResponse = await generateInternalServerErrorRepsonse(error, 'refreshBloomFilter');
        return res.status(500).json(errorResponse);
    }
}

export async function checkPhoneExists(req, res) {
    try {
        const { phoneNumber } = req.query;

        if (!phoneNumber) {
            return res.status(400).json({
                success: false,
                error: 'Phone number is required'
            });
        }

        const mightExist = userSearchBloomFilter.mightExist(phoneNumber);

        return res.status(200).json({
            success: true,
            data: {
                phoneNumber,
                mightExist,
                note: mightExist
                    ? 'Might exist - check database'
                    : 'Definitely does not exist'
            }
        });
    } catch (error) {
        const errorResponse = await generateInternalServerErrorRepsonse(error, 'checkPhoneExists');
        return res.status(500).json(errorResponse);
    }
}

👉 Notice how checkPhoneExists does not immediately hit the DB. It asks the Bloom filter first.

🧠 Service Layer: Core Bloom Filter Logic

This is where the real magic happens.

We maintain four Bloom filters:

emailFilter
phoneFilter
alternateEmailFilter
alternatePhoneFilter

Each filter is initialized with a target capacity and error rate (0.01 = 1%).

import BloomFilter from '../utils/BloomFilter.js';
import User from '../models/user.js';
import logger from '../setup/logger.js';

class UserSearchBloomFilterService {
    constructor() {
        this.emailFilter = new BloomFilter(100000, 0.01);
        this.phoneFilter = new BloomFilter(100000, 0.01);
        this.alternateEmailFilter = new BloomFilter(50000, 0.01);
        this.alternatePhoneFilter = new BloomFilter(50000, 0.01);
        this.isInitialized = false;
        this.lastUpdated = null;
        this.updateInterval = 24 * 60 * 60 * 1000; // 24 hours
    }

🔄 Populating the Filters

On startup, we fetch users in batches and add their identifiers into the filters:

async populateFilters() {
    const batchSize = 1000;
    let offset = 0;
    let hasMoreUsers = true;

    while (hasMoreUsers) {
        const users = await User.query(qb => {
            qb.select('email', 'phone_number', 'alternate_email', 'alternate_phone');
            qb.whereNotNull('email').orWhereNotNull('phone_number');
            qb.limit(batchSize);
            qb.offset(offset);
        }).fetchAll();

        const userList = users.toJSON();
        if (userList.length === 0) {
            hasMoreUsers = false;
            break;
        }

        userList.forEach(user => {
            if (user.email) this.emailFilter.add(user.email);
            if (user.phone_number) this.phoneFilter.add(user.phone_number);
            if (user.alternate_email) this.alternateEmailFilter.add(user.alternate_email);
            if (user.alternate_phone) this.alternatePhoneFilter.add(user.alternate_phone);
        });

        offset += batchSize;
        logger.info(`Processed ${offset} users for bloom filter`);
    }

    logger.info(`Bloom filter population completed. Total users processed: ${offset}`);
}

This ensures all existing users are represented in the filter.

⚡ Lookup Logic

When a phone/email check request arrives:

mightExist(searchKey) {
    if (!this.isInitialized) {
        return true; // fail-safe until initialized
    }

    const normalizedKey = searchKey.toLowerCase().trim();

    return (
        this.emailFilter.mightContain(normalizedKey) ||
        this.phoneFilter.mightContain(normalizedKey) ||
        this.alternateEmailFilter.mightContain(normalizedKey) ||
        this.alternatePhoneFilter.mightContain(normalizedKey)
    );
}

👉 If it returns false, we know for sure the user doesn’t exist.
👉 If it returns true, we query the DB to confirm.

🕒 Auto-Refreshing

To stay fresh with new data, we schedule a 24-hour refresh:

schedulePeriodicUpdate() {
    setInterval(async () => {
        try {
            logger.info('Starting scheduled bloom filter update');
            await this.refresh();
        } catch (error) {
            logger.error('Scheduled bloom filter update failed:', error);
        }
    }, this.updateInterval);
}

This clears and repopulates the filters.

📊 Status Reporting

Finally, we can inspect filter health:

getStatus() {
    return {
        isInitialized: this.isInitialized,
        lastUpdated: this.lastUpdated,
        emailFilterStats: this.emailFilter.getStats(),
        phoneFilterStats: this.phoneFilter.getStats(),
        alternateEmailFilterStats: this.alternateEmailFilter.getStats(),
        alternatePhoneFilterStats: this.alternatePhoneFilter.getStats()
    };
}

🚀 Example Flow

A signup form checks if 1231881971 exists:

Client calls → GET /bloom_filter/check?phoneNumber=<phoneNumber>
Bloom filter says:

❌ Not in set → Return immediately (skip DB).
✅ Might exist → Query DB to confirm.

This cuts DB load massively.

✅ Benefits

O(1) Lookups → Super fast.
Reduced DB Load → Fewer queries.
Scalable → Handles millions of entries.
Fault-Tolerant → Always errs on the safe side (false positives only).

⚠️ Limitations

False Positives → Might say “exists” when it doesn’t.
No Deletion → Standard Bloom filters don’t support removing entries.
Cold Start → Until initialized, returns “might exist” to avoid false negatives.

🌟 Final Thoughts

Bloom Filters are not a database replacement—they’re an optimization layer.
Think of them as a bouncer at the club entrance:

If you’re definitely not on the list, you’re turned away instantly.
If you might be on the list, they’ll double-check inside.

For high-traffic systems, this simple gatekeeping can make a world of difference.

⚡ Deep Dive: How `Promise.all` Works with API & DB Calls in Node.js

Abhinav — Sun, 07 Sep 2025 05:18:42 +0000

1. JavaScript vs Node.js Runtime

JavaScript (V8):
- Single-threaded.
- Executes code top-to-bottom.
- Uses the microtask queue to resolve promises.
Node.js runtime:
- Built on V8 + libuv.
- libuv provides the event loop + thread pool.
- Integrates with the OS networking stack for async I/O.

👉 Node.js achieves concurrency not by adding threads for JS, but by outsourcing work to the OS, DB engines, and libuv workers.

2. How `Promise.all` Works

const p = Promise.all([p1, p2, p3]);

Creates an aggregate promise p.
Subscribes to each input promise (p1, p2, p3).
Tracks results + pending count.
If all resolve → p resolves with an array of results.
If any reject → p rejects immediately.
Resolution runs in the microtask queue, before timers and I/O.

3. API Calls with `Promise.all`

const fetch = require("node-fetch");

async function run() {
  const [r1, r2, r3] = await Promise.all([
    fetch("https://httpbin.org/delay/1"),
    fetch("https://httpbin.org/delay/1"),
    fetch("https://httpbin.org/delay/1")
  ]);
  console.log(await Promise.all([r1.json(), r2.json(), r3.json()]));
}
run();

What happens under the hood:

fetch creates TCP sockets.
OS handles DNS, TCP handshake, TLS.
libuv registers sockets for readiness.
When data arrives → libuv pushes I/O callbacks.
Promises resolve via microtasks.

✅ Concurrency comes from the OS networking stack.

4. DB Queries with `Promise.all`

const { Pool } = require("pg");
const pool = new Pool();

async function run() {
  const [users, orders] = await Promise.all([
    pool.query("SELECT * FROM users LIMIT 5"),
    pool.query("SELECT * FROM orders LIMIT 5")
  ]);
  console.log(users.rows, orders.rows);
}
run();

What happens under the hood:

Queries are written to DB sockets.
PostgreSQL executes them in parallel inside DB worker processes.
Results are streamed back.
libuv notifies event loop.
Promises resolve, aggregated by Promise.all.

✅ Concurrency here is provided by the DB engine.

5. Event Loop & Microtasks

Node.js event loop phases:

Timers → setTimeout, setInterval.
Pending callbacks.
Idle/prepare.
Poll → new I/O events (e.g., socket readiness).
Check → setImmediate.
Close callbacks.

🔑 Between every phase → microtask queue runs first.
That’s why promise resolutions (Promise.all) happen before timers or I/O callbacks.

6. libuv: Networking vs Thread Pool

Networking I/O (fetch, DB sockets):
- Uses OS mechanisms like epoll/kqueue/IOCP.
- No threads involved → highly scalable.
Thread Pool:
- Used when OS doesn’t provide async APIs.
- Examples: fs.readFile, crypto.pbkdf2.
- Default size = 4 (UV_THREADPOOL_SIZE).

👉 API calls & DB sockets rarely use threads — they rely on the OS and external servers for concurrency.

7. Benchmarks: Sequential vs Parallel

🔹 Sequential DB Queries

const { Pool } = require("pg");
const pool = new Pool();

async function sequential() {
  console.time("sequential");
  await pool.query("SELECT pg_sleep(1)");
  await pool.query("SELECT pg_sleep(1)");
  await pool.query("SELECT pg_sleep(1)");
  await pool.query("SELECT pg_sleep(1)");
  await pool.query("SELECT pg_sleep(1)");
  console.timeEnd("sequential");
}
sequential().then(() => pool.end());

Each query sleeps for 1 second.
Run sequentially → ~5 seconds total.

🔹 Parallel DB Queries

const { Pool } = require("pg");
const pool = new Pool();

async function parallel() {
  console.time("parallel");
  await Promise.all([
    pool.query("SELECT pg_sleep(1)"),
    pool.query("SELECT pg_sleep(1)"),
    pool.query("SELECT pg_sleep(1)"),
    pool.query("SELECT pg_sleep(1)"),
    pool.query("SELECT pg_sleep(1)")
  ]);
  console.timeEnd("parallel");
}
parallel().then(() => pool.end());

All queries fired at once.
Each sleeps for 1s → ~1 second total.

🔹 Results

Strategy	Queries	Time
Sequential	5 × 1s	~5s
Parallel	5 × 1s	~1s

🚀 Promise.all gives a 5× speedup by overlapping I/O work.

8. Practical Pitfalls

⚠️ API Rate Limits

Too many concurrent requests → HTTP 429 or throttling. ✅ Use concurrency control (p-limit, Bottleneck).

⚠️ DB Pool Exhaustion

await Promise.all(bigArray.map(() => pool.query("...")));

Exceeds connection pool → queries queue → latency spikes. ✅ Tune pool size + run queries in batches.

⚠️ Error Handling

Promise.all fails fast on the first rejection. ✅ Use Promise.allSettled for resilience.

⚠️ Memory Usage

Large Promise.all → stores all intermediate results in memory. ✅ For very large workloads → stream results instead of batching everything.

9. Alternatives to `Promise.all`

Promise.allSettled → get all results, even if some fail.
Promise.any → resolve on the first success (useful for redundant services).
Promise.race → resolve/reject on the first settled (great for timeouts).

🔑 Final Takeaways

Promise.all coordinates async tasks, it doesn’t make JS parallel.
API concurrency comes from OS networking.
DB concurrency comes from database engines.
Node.js event loop + libuv glue it all together.
Use with care: pool limits, API rate limits, memory usage matter.
Benchmarks show the real-world latency gains, but also highlight scalability risks.

💡 Closing Thought

Promise.all isn’t magic — it’s orchestration.
The “parallelism” you see comes from the OS kernel, libuv, and external systems.

An engineer’s job is not just to know that it “runs concurrently,” but to understand how much concurrency the system can handle before it breaks. That’s what separates toy demos from production-grade systems.

⚡ Deep Dive: How `Promise.all` Works with API & DB Calls in Node.js

Abhinav — Sat, 06 Sep 2025 20:35:05 +0000

1. JavaScript vs Node.js Runtime

JavaScript (V8):
- Single-threaded.
- Executes code top-to-bottom.
- Uses the microtask queue to resolve promises.
Node.js runtime:
- Built on V8 + libuv.
- libuv provides the event loop + thread pool.
- Integrates with the OS networking stack for async I/O.

👉 Node.js achieves concurrency not by adding threads for JS, but by outsourcing work to the OS, DB engines, and libuv workers.

2. How `Promise.all` Works

const p = Promise.all([p1, p2, p3]);

Creates an aggregate promise p.
Subscribes to each input promise (p1, p2, p3).
Tracks results + pending count.
If all resolve → p resolves with an array of results.
If any reject → p rejects immediately.
Resolution runs in the microtask queue, before timers and I/O.

3. API Calls with `Promise.all`

const fetch = require("node-fetch");

async function run() {
  const [r1, r2, r3] = await Promise.all([
    fetch("https://httpbin.org/delay/1"),
    fetch("https://httpbin.org/delay/1"),
    fetch("https://httpbin.org/delay/1")
  ]);
  console.log(await Promise.all([r1.json(), r2.json(), r3.json()]));
}
run();

What happens under the hood:

fetch creates TCP sockets.
OS handles DNS, TCP handshake, TLS.
libuv registers sockets for readiness.
When data arrives → libuv pushes I/O callbacks.
Promises resolve via microtasks.

✅ Concurrency comes from the OS networking stack.

4. DB Queries with `Promise.all`

const { Pool } = require("pg");
const pool = new Pool();

async function run() {
  const [users, orders] = await Promise.all([
    pool.query("SELECT * FROM users LIMIT 5"),
    pool.query("SELECT * FROM orders LIMIT 5")
  ]);
  console.log(users.rows, orders.rows);
}
run();

What happens under the hood:

Queries are written to DB sockets.
PostgreSQL executes them in parallel inside DB worker processes.
Results are streamed back.
libuv notifies event loop.
Promises resolve, aggregated by Promise.all.

✅ Concurrency here is provided by the DB engine.

5. Event Loop & Microtasks

Node.js event loop phases:

Timers → setTimeout, setInterval.
Pending callbacks.
Idle/prepare.
Poll → new I/O events (e.g., socket readiness).
Check → setImmediate.
Close callbacks.

🔑 Between every phase → microtask queue runs first.
That’s why promise resolutions (Promise.all) happen before timers or I/O callbacks.

6. libuv: Networking vs Thread Pool

Networking I/O (fetch, DB sockets):
- Uses OS mechanisms like epoll/kqueue/IOCP.
- No threads involved → highly scalable.
Thread Pool:
- Used when OS doesn’t provide async APIs.
- Examples: fs.readFile, crypto.pbkdf2.
- Default size = 4 (UV_THREADPOOL_SIZE).

👉 API calls & DB sockets rarely use threads — they rely on the OS and external servers for concurrency.

7. Benchmarks: Sequential vs Parallel

🔹 Sequential DB Queries

const { Pool } = require("pg");
const pool = new Pool();

async function sequential() {
  console.time("sequential");
  await pool.query("SELECT pg_sleep(1)");
  await pool.query("SELECT pg_sleep(1)");
  await pool.query("SELECT pg_sleep(1)");
  await pool.query("SELECT pg_sleep(1)");
  await pool.query("SELECT pg_sleep(1)");
  console.timeEnd("sequential");
}
sequential().then(() => pool.end());

Each query sleeps for 1 second.
Run sequentially → ~5 seconds total.

🔹 Parallel DB Queries

const { Pool } = require("pg");
const pool = new Pool();

async function parallel() {
  console.time("parallel");
  await Promise.all([
    pool.query("SELECT pg_sleep(1)"),
    pool.query("SELECT pg_sleep(1)"),
    pool.query("SELECT pg_sleep(1)"),
    pool.query("SELECT pg_sleep(1)"),
    pool.query("SELECT pg_sleep(1)")
  ]);
  console.timeEnd("parallel");
}
parallel().then(() => pool.end());

All queries fired at once.
Each sleeps for 1s → ~1 second total.

🔹 Results

Strategy	Queries	Time
Sequential	5 × 1s	~5s
Parallel	5 × 1s	~1s

🚀 Promise.all gives a 5× speedup by overlapping I/O work.

8. Practical Pitfalls

⚠️ API Rate Limits

Too many concurrent requests → HTTP 429 or throttling. ✅ Use concurrency control (p-limit, Bottleneck).

⚠️ DB Pool Exhaustion

await Promise.all(bigArray.map(() => pool.query("...")));

Exceeds connection pool → queries queue → latency spikes. ✅ Tune pool size + run queries in batches.

⚠️ Error Handling

Promise.all fails fast on the first rejection. ✅ Use Promise.allSettled for resilience.

⚠️ Memory Usage

Large Promise.all → stores all intermediate results in memory. ✅ For very large workloads → stream results instead of batching everything.

9. Alternatives to `Promise.all`

Promise.allSettled → get all results, even if some fail.
Promise.any → resolve on the first success (useful for redundant services).
Promise.race → resolve/reject on the first settled (great for timeouts).

🔑 Final Takeaways

Promise.all coordinates async tasks, it doesn’t make JS parallel.
API concurrency comes from OS networking.
DB concurrency comes from database engines.
Node.js event loop + libuv glue it all together.
Use with care: pool limits, API rate limits, memory usage matter.
Benchmarks show the real-world latency gains, but also highlight scalability risks.

💡 Closing Thought

Promise.all isn’t magic — it’s orchestration.
The “parallelism” you see comes from the OS kernel, libuv, and external systems.

A Technical Deep Dive: The Internals of Database Storage 🔬

Abhinav — Tue, 02 Sep 2025 14:39:23 +0000

Databases power the modern world, but behind the scenes lies a world of ingenious design choices and optimizations. Beyond the basic tables and queries we interact with, databases rely on sophisticated algorithms and data structures to deliver speed, consistency, and reliability at scale. In this deep dive, we’ll peel back the layers and explore the science of database storage.

1. Storage Pages: The Physical Reality 💾

At the lowest level, databases manage data in pages, which are fixed-size I/O units (commonly 8KB). This choice aligns with how operating systems and disk drives interact with data.

Why Pages?

A disk is a physical device with spinning platters and a read/write head. To fetch data, the head must:

Move to the correct cylinder (seek time).
Wait for the right sector to spin into position (rotational latency).

These mechanical delays are orders of magnitude slower than CPU operations. To minimize them, databases read or write entire pages at once, rather than fetching individual rows.

Page Layout

A database page has structure:

Header: metadata (page type, free space, transaction IDs).
Data section: where rows are stored.
Slot array (in slotted pages): pointers to rows at the end of the page, enabling efficient row movement or deletion without shifting the entire block.

📊 Illustration: Page Layout

+---------------------+
| Page Header         |
+---------------------+
| Row Data (variable) |
| Row Data (variable) |
| Row Data (variable) |
+---------------------+
| Slot Array (pointers)|
+---------------------+

2. Indexes: The B-Tree Workhorse 🌳

Indexes are the backbone of query performance, and the most common structure is the B+ Tree.

Structure

Unlike binary trees, B-Trees have high fanout (many children per node). Each node corresponds to a disk page. This keeps the tree short and wide, reducing the number of I/O operations needed.

Internal nodes: contain keys and pointers (navigation).
Leaf nodes: contain actual row pointers and are linked in a sorted doubly linked list.

📊 Illustration: B+ Tree

           [50]
         /      \
   [20,30]      [60,80]
   /  |  \      /   |   \
[10] [25] [35] [55] [65] [90]

Leaf nodes are linked -> [10] <-> [25] <-> [35] <-> [55] <-> [65] <-> [90]

Why B+ Trees Are Powerful

Point Lookups: Traversing from root to leaf typically requires just 3–4 disk reads, even in massive tables.
Range Scans: Once the starting leaf is found, the database follows the linked list across leaf nodes for fast sequential access.

This flexibility is why B+ Trees are the default index type in most relational databases, supporting queries like =, <, >, BETWEEN, and prefix-based LIKE.

3. Write-Ahead Logging (WAL): The ACID Foundation ⚛️

Ensuring Atomicity and Durability of transactions requires careful design. That’s where Write-Ahead Logging (WAL) comes in.

How WAL Works

Before modifying a data page, the database writes a redo log record to the WAL.
Once the WAL record is safely on disk, the transaction is considered committed.

📊 Illustration: WAL Flow

Transaction -> WAL (log record) -> Commit Success
            -> Data Page (later written)

Crash Recovery

Redo Phase: Reapply all committed changes from the WAL since the last checkpoint.
Undo Phase: Roll back any incomplete transactions.

This decoupling of transaction commit from physical data writes allows high throughput without sacrificing safety.

4. Concurrency with MVCC: The Snapshot Approach 📸

Concurrency is a major challenge in databases. Traditional locks can lead to contention and deadlocks. Multi-Version Concurrency Control (MVCC) offers a smarter approach.

How MVCC Works

Each row stores xmin (creating transaction ID) and xmax (deleting/updating transaction ID).
A transaction sees only rows where:
- xmin < transaction_id
- xmax is unset or greater than the transaction’s ID.

📊 Illustration: Row Versions

Row A (xmin=100, xmax=200)
Row B (xmin=150, xmax=∞)

Transaction 180 sees Row A
Transaction 250 sees Row B

This means readers never block writers, and writers don’t block readers. Each transaction gets a snapshot of the database as of its start time.

Benefits of MVCC

Readers don’t wait for writers: Queries can run without being blocked by updates.
Writers don’t block readers: Updates and deletes proceed while readers still see a consistent snapshot.
High concurrency: Makes it easier to scale workloads with many parallel transactions.

Challenges of MVCC

Storage bloat: Multiple row versions can increase disk usage.
Vacuuming / Garbage Collection: Old row versions must eventually be cleaned up.
Complex transaction management: Determining visibility rules requires careful bookkeeping of transaction IDs.

Real-World Use

PostgreSQL: Implements MVCC with transaction IDs (xmin, xmax) and requires VACUUM to reclaim space.
Oracle: Uses UNDO segments to maintain old row versions.
MySQL InnoDB: Maintains undo logs and hidden system columns to support MVCC.

MVCC provides a balance between consistency and performance, making it the preferred model in modern relational databases.

5. Row vs. Column Storage: The OLTP vs. OLAP Trade-off 📊

The choice between row and column storage determines a database’s strengths.

Row-Oriented (OLTP)

Stores all values of a row contiguously.
Example: (CustomerID: 123, Name: John Smith, City: NYC)
Optimized for frequent inserts, updates, and whole-row reads.
Ideal for transactional systems like banking or e-commerce.

Column-Oriented (OLAP)

Stores all values of a column contiguously.
Example: (CustomerID: 123, 124, 125...), (City: NYC, LA, SF...)
Optimized for analytical queries scanning few columns across many rows.
Enables compression (similar values compress well).
Perfect for data warehouses and reporting systems.

📊 Illustration: Row vs Column

Row Store:
[123 | John Smith | NYC]
[124 | Jane Doe   | LA ]

Column Store:
CustomerID: [123,124]
Name:       [John, Jane]
City:       [NYC, LA]

Closing Thoughts ✨

What feels like “magic” when running a fast query or recovering from a crash is really the result of decades of careful engineering:

Pages minimize slow disk I/O.
B+ Trees balance speed and flexibility.
WAL ensures durability without blocking.
MVCC delivers smooth concurrency with snapshots.
Row vs. Column storage tailors systems to different workloads.

The next time we query a database, we’ll know the science humming beneath the surface — turning spinning platters and raw bytes into reliable, high-performance systems.

Android 15 and Safe Areas: Why Apps Feel Different Now

Abhinav — Sun, 31 Aug 2025 08:09:09 +0000

Getting to work with a team of app developers gives you crazy insights about tech. Recently, Android 15 (API level 35) quietly changed something we’ve all experienced but rarely thought about: where apps are allowed to live on the screen.
With this release, Google has made edge-to-edge layouts the default. That means apps no longer sit neatly under the status bar or above the navigation bar—they now stretch into every pixel of your display.

This is about more than looks. It changes how apps interact with the parts of the screen traditionally reserved for system elements, also known as safe areas.

What Are Safe Areas?

Safe areas are screen zones guaranteed to be free from system UI elements. Think of them as invisible boundaries around:

The status bar at the top (time, battery, network icons)
The navigation bar or gesture area at the bottom
Cutouts or notches for cameras and sensors

In earlier Android versions, apps automatically avoided these regions. Even if you weren’t thinking about it, your phone made sure a “Send” button didn’t hide behind the navigation bar.

What Changed in Android 15

With Android 15, Google flipped the default:

Edge-to-edge by default
Apps now extend behind the status bar, navigation bar, and cutouts.
Cutouts are always allowed
Even if an app didn’t explicitly request it, content may appear under a notch or hole-punch camera.
No more automatic padding
The system doesn’t hold back space for apps anymore. Developers need to consciously respect safe areas if they don’t want text, buttons, or images to get hidden.

In short, the system stopped babysitting layouts. Apps now own the full canvas—but with that freedom comes responsibility.

Why This Matters

For users:

Apps feel more immersive—photos, videos, and maps stretch to the very edges.
The overall look feels cleaner and modern.

For apps:

Poorly adapted apps might show text under the battery icon, or have buttons blocked by the navigation bar.
Well-designed apps now look sleek, balanced, and edge-to-edge.

This shift echoes Apple’s move during the iPhone X era, when apps had to learn to live around the notch.

Why Google Did This

To create a consistent full-screen design language across Android.
To encourage modern UI practices—instead of legacy layouts frozen in older styles.
To take advantage of today’s larger, higher-resolution displays.

It’s part of Android’s ongoing push toward immersive visuals, where every pixel matters.

The Big Picture

Android 15 changes the role of safe areas:

Before: the system managed them for you.
Now: apps must be aware of them.

For most people, this means apps will gradually look sharper and more immersive as developers adapt. For developers, it’s a wake-up call to modernize layouts.

Want to Dive Deeper?

Here are some excellent resources that explain the shift:

✨ In short: Android 15 makes apps live in the entire screen space. It feels immersive and future-ready, but also demands that apps adapt to safe areas thoughtfully. The result? A more modern Android experience for all of us.