Forem: Pau Dang

Stop Wasting Time on Boilerplate: Real-world Kafka & PostgreSQL Demo in 8 minutes

Pau Dang — Tue, 07 Apr 2026 13:00:00 +0000

After releasing the v2.0.0 Web UI for Node.js Quickstart Generator, the most common question was: "How does it handle real-world complexity?"

So, I decided to record a full, 8-minute implementation demo building a Payment Service from scratch.

📺 Watch: UI to Production Code in 8 Minutes

https://youtu.be/PmmxJLloZ1Q

🛠️ The Tech Stack (Zero-Prompt Setup)

Instead of answering 20 CLI prompts, I used our new Web Configurator to generate this exact stack:

Language: TypeScript
Architecture: Clean Architecture (Domain, UseCase, Infra)
Database: PostgreSQL
Caching: Redis
Messaging: Kafka
Security: Snyk Verified

🏗️ Clean Architecture in Action

The video shows exactly how the folder structure reflects a production-grade system:

src/domain: Pure entities with no dependencies.
src/usecases: Where the transaction logic lives.
src/infrastructure: Concrete implementations for Postgres connections and Kafka producers.

📡 Live Kafka Flow

The "Wow" moment is at 05:00 in the video. We trigger a REST API call that producers a Kafka event, which is then picked up by a separate consumer.

Zero configuration required.
Pre-mapped events.
Ready for microservices.

🛡️ Enterprise-Grade Security

We don't skip security. I ran npm run security:check in the video to show how Snyk is integrated from the start. Clean code is nothing if it isn't secure.

💻 Try it yourself

Want to generate the exact same project as in the video? Run this:

npx nodejs-quickstart-structure@latest init -n "payment-service-nodejs" -l "TypeScript" -a "Clean Architecture" -d "PostgreSQL" --db-name "payment-db" -c "Kafka" --caching "Redis" --ci-provider "GitHub Actions" --include-security

Check out our GitHub Repo and let's end boilerplate fatigue together! 🌟

Introducing Node.js Quickstart Generator v2.0.0: Automated Clean Architecture with a Sleek New UI

Pau Dang — Mon, 06 Apr 2026 12:24:35 +0000

Hi everyone,

How many times have you set up a new Node.js microservice by copying a folder from your last project? We all do it out of necessity. But after releasing v1.1.0, the scale of "boilerplate fatigue" became clear: over 4,000 developers joined the journey in just one month.

That growth sparked a realization: we needed more than just a template. Previously known as nodejs-quickstart-structure, we have officially evolved into a full-scale Generator in v2.0.0.

I built this tool to help ourselves: Why can't our starting points be enterprise-ready by default?

🎥 The Evolution: From CLI (v1.1.0) to Web UI (v2.0.0)

How it started: A powerful CLI that 4,000 of you loved (v1.1.0).

https://www.youtube.com/watch?v=Cxbb54T0uo8
How it’s going: A full-blown Web UI with 1:1 structure parity (v2.0.0).

Introducing: Node.js Quickstart Generator 2.0.0

I built this tool to solve my own frustration: Why can't I just have a professional-grade starting point in one command?

✨ What’s in the Box?

Flexible Architecture: Toggle between MVC and Clean Architecture.
Modern Stack: Pick JavaScript or TypeScript.
Communication Protocols: Built-in support for REST, GraphQL (Apollo), and Kafka (Event-driven).
Caching Layer: Pre-configured Redis or In-memory cache.
Enterprise Standards:
- Multi-stage Dockerfiles.
- Snyk & SonarCloud security hardening.
- Global error handling (ApiError, NotFoundError, etc.).
- Jest & Supertest with 80% unit test coverage out of the box.

🌐 Next-Gen Web UI Configurator (v2.0.0)

Type no more! We have completely evolved the configuration experience in our new Web UI Configurator.

It includes a Real-time Folder Simulation. As you toggle between MVC and Clean Architecture, you see exactly what the project will look like. Once you're happy, just copy the Zero-Prompt CLI command and run it in your terminal. No more prompts!

🦾 AI-Native Foundation

We designed v2.0.0 for the future. If you use Cursor or other AI coding agents, our built-in .cursorrules will make your AI assistant 10x smarter about your specific architecture. No more hallucinations about where controllers or repos belong!

🎯 Get Started NOW

npx nodejs-quickstart-structure@latest init

That’s it. No global install required.

🗺️ Road to 10k Downloads

We just crossed 4,000+ downloads and want to reach 10k by the end of the year. If you love open-source tools that actually save you time, please:

Give us a ⭐ on GitHub.
Share this with your team.
Drop a comment below—what feature should we add next? (gRPC? NestJS-style?)

24 Hours of Chaos: Saving My Open Source Project from a Supply Chain Attack (plain-crypto-js)

Pau Dang — Wed, 01 Apr 2026 01:01:37 +0000

Hello world,

I'm a Senior SE. Today, I want to share a "battle-tested" experience that just happened to my open-source project: nodejs-quickstart-structure.

This isn't just about code; it’s a lesson in Incident Response when facing professional malware designed to hijack npm, GitHub, and sensitive developer credentials.

1. The Threat: Axios & plain-crypto-js

While developing version v2.0.0, I fell victim to a Typosquatting attack. A malicious package or a "shell" dependency injected malware into my local environment.

The Suspect: Linked to the plain-crypto-js incident (a malware variant targeting devs using Axios).
The Behavior: It didn't just break my system; it silently exfiltrated:
- Browser Cookies: Hijacking active sessions for Gmail, GitHub, and LinkedIn.
- SSH Keys: Gaining unauthorized access to push code to repositories.
- npm Tokens: Attempting to publish malicious releases under my name.

2. 0h00: Detection & Containment

Immediately after noticing suspicious logs and file modifications, I followed the "Security Textbook" or you can check at Axios npm Package Compromised: Supply Chain Attack Delivers Cross-Platform RAT:

Deleted Local Repos: Wiped the execution environment of the malware.
Revoked All Sessions: Used a clean device (mobile) to remotely sign out of Google, GitHub, Microsoft, and LinkedIn.
Untrusted Devices: Removed my current machine from the "Trusted Devices" list of all critical accounts.

3. The Battle for npm (The Support Battle)

The worst-case scenario: The attacker hijacked the session and invalidated my 2FA (my stored Recovery Codes returned Invalid).

I immediately contacted npm Support:

Ticket ID: 4223695 was created.

The Strategy: Providing proof of ownership through my GitHub account (which I still control) and the project's long-standing commit history.

4. The Decision: Eradication (Wipe & Rebuild)

As an Architect, I know that if an OS is compromised by a Rootkit/Trojan, no antivirus can guarantee a 100% clean state. The only solution: Wipe & Rebuild.

The Method: Reset PC > Remove everything > Cloud download Windows.
Why Cloud Download? To ensure a fresh installation image directly from Microsoft, avoiding any malware lurking in the local Recovery partition.

5. Lessons Learned for Developers

Dependency Vigilance: Always double-check new packages, especially those with names similar to popular libraries.
2FA is Not Enough: Attackers can bypass 2FA via Session Hijacking. Always be ready to Revoke Sessions remotely.
Offline Recovery Codes: Don't just store them on your computer. Print them or use a decoupled password manager.
Incident Response Mindset: When hacked, stay calm and follow: Containment -> Asset Protection -> Eradication -> Recovery.

Currently, I am in the process of restoring a "sterile" environment to finalize v2.0.0 for nodejs-quickstart-structure. You can check out the v2.0.0 beta details here:

Next gen Web UI - Browser Generator

The project will return with a higher security standard. I hope this story helps fellow developers protect their "digital children"!

How I Built a Node.js Generator with 1,680+ Combinations, Clean Architecture or MVC (Kafka)

Pau Dang — Thu, 26 Mar 2026 10:01:34 +0000

Hi all,

We’ve all been there: starting a new Node.js microservice from scratch. You spend 4 hours setting up Express, another 2 hours arguing over folder structure (MVC or Clean Architecture?), 3 hours configuring Prisma or Mongoose, and half a day trying to get a Kafka KRaft container to talk to your local app.

By the time you're ready to write your first line of business logic, you're already exhausted.

I decided to solve this once and for all. I built nodejs-quickstart-structure—a CLI that doesn't just give you a "hello world," but scaffolds a production-ready engine tailored to your exact needs.

The Problem: The "Boilerplate exhaustion"

Most generators give you a fixed stack. But in the real world, requirements vary:

"This needs to be a simple MVC app."
"This needs to be a Clean Architecture domain-driven service."
"We need GraphQL, not REST."
"We need Kafka for event-driven messaging."

The Solution: 1,680+ Scenarios in One CLI

The core of this tool is its flexibility. By combining different technologies, it supports over 1,680 unique project combinations:

Languages: TypeScript (Recommended) / JavaScript.
Architectures: MVC or Clean Architecture.
Databases: MySQL, PostgreSQL, MongoDB, or None.
Communications: REST APIs, GraphQL, or Kafka (Event-Driven).
Caching: Redis or Memory Cache.
Security: Hardened with Helmet, HPP, and automated Snyk/SonarCloud configs.
CI/CD: Ready-to-use workflows for GitHub Actions, GitLab CI, or Jenkins.

Why "Clean Architecture"?

For many projects, MVC starts fast but becomes a nightmare to test. I’ve implemented a robust Clean Architecture template where:

Domain: Pure business logic (Entities/Use Cases).
Infrastructure: Database, Messaging (Kafka Client), Caching.
Interfaces: Controllers, Express Routes, GraphQL Resolvers.

This separation ensures your business logic doesn't care if you're using MongoDB or PostgreSQL.

AI-Native: Coding in 2026

I realized that if the folder structure is standard, AI tools (like Cursor, ChatGPT, or Gemini) can understand the codebase 10x faster.
Every project generated includes:

.cursorrules: Pre-configured rules so Cursor knows exactly how to add new features following your chosen architecture.
prompts/: A library of "Agent Skills" to help you generate Use Cases or Repositories without breaking patterns.

Try it out

You don't even need to install it globally. Just run:

npx nodejs-quickstart-structure init

The CLI will walk you through the setup. Within seconds, you'll have a project with 80%+ unit test coverage, ESLint/Prettier configured, and a Docker Compose file that actually works.

Checking the official docs:
👉 Official Documentation

Check out the repo and give it a ⭐ if it helps your workflow:
👉 GitHub: paudang/nodejs-quickstart-structure

I'd love to hear your feedback! What's the one thing you always struggle with when starting a new Node project?

Stop Writing Flaky Tests: The Ultimate Node.js Testing Strategy (Unit + E2E)

Pau Dang — Mon, 23 Mar 2026 13:00:00 +0000

Hi DEV community,

If you are a Backend Developer working with Node.js, you have likely experienced the dreaded scenario: "It passes on my machine, but randomly fails on the CI/CD pipeline."

This phenomenon is known as Flaky Tests. It usually stems from writing End-to-End (E2E) tests that share database states across test files, or due to network and infrastructure services (Redis, Kafka) not being fully initialized when the test begins.

Today, I’m going to share the complete testing architecture and lessons I learned while building the automation framework nodejs-quickstart-structure. We will solve the core problem: How to build blazing fast Unit Tests and completely deterministic E2E Tests.

1. The Crisis in 90% of Projects

Many teams implement testing "half-heartedly":

Unit Tests: Dependencies like the Database or Redis aren't mocked, causing the tests to drag on because they wait for network I/O.
E2E Tests: Developers use their local dev database to run E2E suites. Test A creates a User, Test B checks the total number of Users. If they run in a different order, the test suite explodes! Some even use conditionals in tests: if (statusCode == 404) expect(404) else expect(201).

This is a massive Anti-pattern! A test must strictly return exactly one predictable result based on static inputs.

2. The "Big Tech" Strategy: Draw a Hard Line

To fix this, you must strictly delineate your boundaries:

Unit Tests (Fast & Isolated)

Goal: Verify Business Logic (Use cases, Services, Domain).
Rule: MOCK EVERYTHING. No real database connections, no external APIs, no touching Redis or Kafka.
Speed: Thousands of test cases should execute in less than 2 seconds.

E2E Tests (Black-box & Automated Infra)

Goal: Verify the entire request flow (Route -> Controller -> Usecase -> Repo -> DB -> Response).
Rule: Use the REAL Database, Redis, and Kafka (spinned up via isolated Docker Containers or Testcontainers).
Characteristics: Data must be TRUNCATED/Teared down before or after each test suite to guarantee a "clean room" environment. It runs slower, but absolute correctness is guaranteed.

3. The Recipe for Perfect E2E Tests

To prevent E2E tests from interfering with the developer's local development environment, follow these steps and source demo nodejs-service-redis-kafka:

Step 1: Fully isolate `jest.config.js`

Do not share your Unit test configurations with E2E. Create a dedicated jest.e2e.config.js with a higher testTimeout (e.g., 30 seconds to allow databases to boot).

module.exports = {
  ...require('./jest.config'),
  testMatch: ['<rootDir>/tests/e2e/**/*.test.ts'],
  testPathIgnorePatterns: ['/node_modules/'],
  testTimeout: 30000, 
  clearMocks: true
};

Step 2: Use Node.js Scripts to Manage Docker Lifecycle

Instead of forcing developers to manually type docker-compose up before running tests, write an automated orchestration script:

Assign a dedicated port (PORT=3001 instead of 3000) to avoid Dev Server collisions.
execSync('docker-compose up -d db redis kafka').
Use the wait-on npm package to poll the healthcheck until dependencies are fully green.
Run npm run test:e2e:run.
Clean up gracefully: execSync('docker-compose down').

// Wait for dependencies to prevent "Flaky connections"
execute(`npx wait-on http-get://127.0.0.1:${TEST_PORT}/health -t 120000`);
execute('jest --config ./jest.e2e.config.js');

Step 3: Fix Kafka's "read ECONNRESET" Locally

The most common issue when testing Kafka in a local E2E run is that the tests run on the host network while Kafka is stuck in the Docker bridged network.

The Fix: Explicitly map the PLAINTEXT_HOST listener to a dedicated port (e.g., 9093):

# docker-compose.yml
kafka:
  ports:
    - "9093:9093" # Host mapping
  environment:
    - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:9093
    - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,PLAINTEXT_HOST://:9093,CONTROLLER://:9094

Then, in your .env.test, simply point KAFKA_BROKER=localhost:9093.

Step 4: Write "Iron-clad" Assertions

Eliminate loose assertions. Because your database is wiped clean (or relies on random seeds like Date.now()), the output status must be absolutely rigid!

it('should create a user successfully via REST', async () => {
    const uniqueEmail = `test_${Date.now()}@example.com`;
    const response = await request(SERVER_URL)
        .post('/api/users')
        .send({ name: 'Test User', email: uniqueEmail });

    // Strictly expect a 201 Created
    expect(response.statusCode).toBe(201);
});

4. Conclusion

Shifting to an isolated, automated Docker test strategy will cost you 1-2 days of initial setup infrastructure work. But in return, it brings absolute peace of mind to the team as the codebase scales.

If you find this setup process too tedious, you can simply grab the exact folder structures, Docker automation scripts, and Jest configurations that I’ve already pre-configured out of the box in my open-source CLI generator:
👉 nodejs-quickstart-structure on GitHub

Drop a star (⭐) if you find it helpful! Happy coding, and here's to never seeing Test Failed randomly in GitHub Actions again!

Master Caching Patterns: A Clean Architecture Guide with AI-Native Tooling

Pau Dang — Thu, 19 Mar 2026 15:02:06 +0000

In high-performance systems, caching is the ultimate "lifesaver" for reducing database load and optimizing response times. However, choosing the right pattern (Cache-Aside, Write-Through, etc.) depends heavily on your specific use case. In this article, I’ll walk you through a hands-on demo project that showcases 5 essential caching patterns, built with Clean Architecture and the power of an AI-Native tool: nodejs-quickstart-structure.

🚀 Why Caching?

When scaling applications, the database often becomes the bottleneck. Caching strategically places frequently accessed data in memory (like Redis) to bypass slow disk I/O. But not all caching is equal. To understand the nuances, I built a showcase service from scratch.

To skip the boilerplate and focus on the patterns, I used nodejs-quickstart-structure, a CLI tool designed for AI-ready, structured development.

📂 5 Caching Patterns You Must Know

Here is a breakdown of the 5 patterns implemented in this demo:

1. Cache-Aside (Lazy Loading)

Purpose: The application manages the cache. Data is only loaded into the cache when specifically requested.
When to use: Best for systems with Read >> Write ratios (e.g., User Profiles).
Pros: Memory efficiency; cache only contains data that is actually needed.

2. Read-Through

Purpose: Offloads data-fetching logic to the Cache Provider. The app simply calls "Get" from the cache.
When to use: To keep the business logic (Use Case) clean and decouple from the database.
Pros: Extremely clean business code.

3. Write-Through

Purpose: Data is written to both the Cache and the Database simultaneously.
When to use: When Strong Consistency is required.
Pros: Minimal data inconsistency risk.

4. Write-Around

Purpose: Data is written directly to the DB, and the corresponding cache entry is invalidated (deleted).
When to use: When the written data might not be read again immediately.
Pros: Prevents "polluting" the cache with data that won't be reused soon.

5. Write-Back (Write-Behind)

Purpose: Data is written to the Cache first; the DB update happens asynchronously in the background.
When to use: For exceptionally high write performance (e.g., Logging, Real-time Metrics).
Pros: Near-instant write response.

🛠️ Technical Implementation (Step-by-Step)

Follow this roadmap to see how I build a production-ready demo using Clean Architecture:

1. Project Initialization

Bootstrap the project using the AI-Native CLI:

npx nodejs-quickstart-structure@latest init

Selection Guide:

Project name: nodejs-service-caching-pattern
Architecture: Clean Architecture
Database: MySQL
Caching: Redis
Communication: REST APIs

2. Defining Domain Interfaces

Abstract the caching and repository logic to ensure true decoupling.

ICacheService.ts

export interface ICacheService {
  get<T>(key: string): Promise<T | null>;
  set(key: string, value: unknown, ttl?: number): Promise<void>;
  del(key: string): Promise<void>;
  getOrSet<T>(key: string, fetcher: () => Promise<T>, ttl?: number): Promise<T>;
}

IUserRepository.ts

3. Infrastructure Layer

Implementation for Redis and Sequelize (MySQL).

4. Database Seeding: Generate 1,000 dummy users for demo purposes.

V20260319__seed_1000_users.sql

5. Implementing the Use Cases

This is where the pattern logic lives.

Read-Through Example: getUserInfoReadThrough.ts

// The cache provider handles DB fetching upon miss
return await this.cacheService.getOrSet<User | null>(cacheKey, async () => {
    return await this.userRepository.findById(id);
}, 3600);

Write-Back Example: updateUserWriteBack.ts

// Update cache immediately, DB updates in background
await this.cacheService.set(cacheKey, updatedUser, 3600);
this.asyncDatabaseUpdate(updatedUser);

You can check sample code at here:

6. API & Documentation

🔍 How to Test & Verify

Once deployed, you can trigger these endpoints to verify the internal logic:

1. Test Cache-Aside / Read-Through (Read Path)

First Call (Cold Cache): curl -X GET http://localhost:3000/api/demo/cache-aside/1
- Internal: App checks Redis (Miss) -> Queries MySQL -> Updates Redis -> Returns.
Subsequent Call (Warm Cache): Repeat the command.
- Internal: App checks Redis (Hit) -> Returns immediately with zero DB overhead.

2. Test Write-Through (Sync Write)

Trigger: curl -X POST http://localhost:3000/api/demo/write-through -H "Content-Type: application/json" -d '{"name": "Performance", "email": "perf@test.com"}'
- Internal: Service writes to MySQL AND Redis simultaneously.

3. Test Write-Around (Clean Write)

Trigger: curl -X POST http://localhost:3000/api/demo/write-around -H "Content-Type: application/json" -d '{"name": "Guest", "email": "guest@test.com"}'
- Internal: Writes to MySQL, then calls DEL to invalidate the Redis key. The next read will be a Miss to load the fresh data.

4. Test Write-Back (Async Write)

Trigger: curl -X PUT http://localhost:3000/api/demo/write-back/1 -H "Content-Type: application/json" -d '{"name": "Fast Update"}'
- Internal: Updates Redis immediately (super fast response). The DB update happens after a short delay asynchronously.

💡 Final Thoughts on AI-Native Tooling

What makes nodejs-quickstart-structure special is not just the speed of initialization, but its AI-Native nature. With .cursorrules and pre-built prompts, it ensures that your project maintains Clean Architecture and consistent coding standards automatically from development to production.

Full Source Code: GitHub Repository

I hope this guide helps you choose the right caching pattern for your next project

Why Your Docker Container Works on Windows but Fails on Linux: The Case-Sensitive Naming Nightmare

Pau Dang — Thu, 19 Mar 2026 00:00:00 +0000

Hi Everyone,

Have you ever faced that frustrating moment: Your code runs perfectly on your local machine (Windows/macOS), but the moment you containerize it with Docker or deploy it to a Linux server, everything breaks with a cryptic Module not found error?

The culprit usually isn't your logic—it’s a "sweet lie" told by your operating system.

1. The Kernel-Level Reality Check

The trouble starts with how different Operating Systems handle their File Systems:

Windows (NTFS): Microsoft prioritizes user convenience. To Windows, User.service.ts and user.service.ts are semantically the same. Therefore, NTFS is designed to be Case-Insensitive.
Linux (Ext4/XFS): The philosophy of Linux (and Unix) is absolute precision. In ASCII, 'U' (65) and 'u' (117) are two completely different entities. Linux treats User.ts and user.ts as two distinct files that can coexist in the same folder.

2. The Docker "Trap" on Local Machines

Docker on Windows or macOS doesn't actually run directly on those kernels. It runs inside a Lightweight Linux VM.

When you mount (bind mount) your code from a Windows host (Case-Insensitive) into that Linux VM (Case-Sensitive):

The Mistake: You name your file UserMapper.ts but write import { UserMapper } from './usermapper'.
The Local Pass: Windows tells the engine: "Sure, I know what you mean, here is the file."
The Docker Crash: The Linux environment inside Docker yells: "I don't see any file named usermapper (lowercase) here!"

The Result: Your system crashes at runtime or fails the build process immediately.

3. Vaccinating Your Node.js Projects

Instead of relying on human memory, we should use tools to enforce consistency. In my Open Source project, nodejs-quickstart-structure, I’ve implemented a 3-layer defense system:

Layer 1: ESLint Enforcement (Stop it at the source)

We use the eslint-plugin-import plugin to flag errors directly in VS Code if your import path doesn't match the actual filename 100%.

"rules": {
  "import/no-unresolved": "error"
}

Layer 2: CI/CD Pipeline (The Ultimate Gatekeeper)

Our GitHub Actions run on Ubuntu (Linux). If you accidentally push a naming mismatch, the pipeline will fail during the test suite, preventing broken code from ever reaching Production.

Layer 3: Standardized Naming Conventions

Stick to a strict naming convention across the entire project. Whether it's kebab-case or PascalCase, consistency is your best friend.

Final Thoughts

Linux environments don't create bugs; they simply expose the inconsistencies that Windows hides from you. Synchronizing your environment from Local to Production is a hallmark of a Senior Developer.

If you’re looking for a Node.js boilerplate that is pre-configured to be "immune" to these environment issues (along with Kafka, Redis, and Clean Architecture), feel free to check out my project:

👉 GitHub: nodejs-quickstart-structure

👉 NPM: npx nodejs-quickstart-structure init

I hope this saves you a few hours of meaningless debugging! Feel free to drop a ⭐ if you find the project helpful!

15 Minutes to "Ship It": (Clean Architecture + REST API + Kafka + Docker & CI/CD) From Zero to Production with Node.js

Pau Dang — Mon, 16 Mar 2026 02:02:12 +0000

Starting a new Node.js project often involves tedious repetitive tasks: scaffolding directory structures, setting up Express, configuring database connections, managing migrations, and integrating messaging systems like Kafka. This "boilerplate phase" can eat up hours of your initial development time.

Today, I’ll show you how to go from zero to a production-ready environment in minutes. We will build a high-performance Node.js service using Clean Architecture, TypeScript, MySQL, Flyway for database migrations, Kafka for real-time event-driven messaging, Docker Compose for orchestration, and GitHub Actions for CI/CD.

Let’s dive in!

🎯 "Ready-to-Run" Source Code for You:
Instead of manually copying snippets, I’ve packaged the entire source code for this article into a production-grade template on GitHub. This project has already reached 3,000+ downloads and is being used by developers for real-world services.

🔗 Repo: paudang/nodejs-clean-rest-kafka

(Just git clone, run docker-compose up -d, and you're live!)

Step 1: Initialize Project & Install Dependencies

First, let's create the project directory:

mkdir nodejs-clean-rest-kafka
cd nodejs-clean-rest-kafka
npm init -y

Install the essential production libraries:

npm install express cors helmet hpp express-rate-limit dotenv morgan kafkajs sequelize mysql2 winston

Install development dependencies:

npm install -D typescript @types/node @types/express @types/cors @types/morgan ts-node tsconfig-paths tsc-alias jest ts-jest @types/jest

Initialize tsconfig.json with Path Aliases (@/*):

{
  "compilerOptions": {
    "target": "es2020",
    "module": "commonjs",
    "outDir": "./dist",
    "rootDir": "./src",
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "baseUrl": ".",
    "paths": {
      "@/*": ["src/*"]
    }
  },
  "include": ["src/**/*"]
}

Step 2: Architecting for Scale (Clean Architecture) 🏗️

Using Clean Architecture ensures your codebase remains decoupled and highly testable. We divide the source code into distinct layers:

mkdir -p src/domain src/usecases src/interfaces src/infrastructure src/utils

src/domain: Core business entities and logic (framework-independent).
src/usecases: Application-specific business rules (The "Interactors").
src/interfaces: Adapters such as Controllers and HTTP Routes.
src/infrastructure: Technical details like Database connections, Kafka clients, and Loggers.
src/utils: Shared utility functions.

This separation allows you to swap your database or framework without touching the core business logic.

Step 3: Event-Driven Messaging with Kafka 🚀

In a microservices architecture, Kafka acts as the "heartbeat" for asynchronous communication. We’ll build a KafkaService using the Connection Promise pattern to ensure the Producer is fully connected before sending messages, preventing data loss during startup.

Inside src/infrastructure/messaging/kafkaClient.ts:

import { Kafka, Producer } from 'kafkajs';

export class KafkaService {
    private producer: Producer;
    private isConnected = false;
    private connectionPromise: Promise<void> | null = null;

    constructor() {
        const kafka = new Kafka({ clientId: 'user-service', brokers: ['localhost:9092'] });
        this.producer = kafka.producer();
    }

    // "Connection Promise" ensures we only connect once efficiently
    async connect() {
        if (this.connectionPromise) return this.connectionPromise;

        this.connectionPromise = (async () => {
            await this.producer.connect();
            this.isConnected = true;
            console.log('[Kafka] Producer connected successfully');
        })();
        return this.connectionPromise;
    }

    async sendEvent(topic: string, action: string, payload: any) {
        await this.connect(); // Always wait for readiness
        await this.producer.send({
            topic,
            messages: [{ value: JSON.stringify({ action, payload, ts: new Date() }) }],
        });
        console.log(`[Kafka] Triggered ${action} to ${topic}`);
    }
}

export const kafkaService = new KafkaService();

Step 3.5: Scaling Consumers with Clean Interfaces 🛠️

Managing dozens of event types can quickly become messy. We use Abstract Base Classes and Schema Validation to keep consumers organized:

1. BaseConsumer (The Blueprint)

At src/interfaces/messaging/baseConsumer.ts, we define a template for all consumers. It handles JSON parsing and error logging, so subclasses can focus solely on business logic:

export abstract class BaseConsumer {
  abstract topic: string;
  abstract handle(data: unknown): Promise<void>;

  async onMessage({ message }: EachMessagePayload) {
      const rawValue = message.value?.toString();
      if (!rawValue) return;
      const data = JSON.parse(rawValue);
      await this.handle(data);
  }
}

2. Schema Validation (The Contract)

Using Zod at src/interfaces/messaging/schemas/userEventSchema.ts, we define the contract between Producer and Consumer. This ensures type safety across the wire.

3. WelcomeEmailConsumer (The Implementation)

Logic that triggers an email when a USER_CREATED event is received:

export class WelcomeEmailConsumer extends BaseConsumer {
  topic = 'user-topic';

  async handle(data: unknown) {
    const result = UserEventSchema.safeParse(data);
    if (result.success && result.data.action === 'USER_CREATED') {
      console.log(`[Kafka] 📧 Sending welcome email to ${result.data.payload.email}...`);
    }
  }
}

Step 4: Database Version Control with Flyway

Manual database changes are a nightmare in production. Flyway manages your schema versioning through simple SQL files.

Example V1__Create_Users_Table.sql:

CREATE TABLE users (
    id INT AUTO_INCREMENT PRIMARY KEY,
    name VARCHAR(255) NOT NULL,
    email VARCHAR(255) NOT NULL UNIQUE,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

Step 5: Clean UseCases & Controllers

1. UseCase (The Interactor)

Decoupled logic at src/usecases/createUser.ts:

export const createUserUseCase = async (name: string, email: string) => {
    const user = await userRepository.save({ name, email }); // Repository abstraction
    await kafkaService.sendEvent('user-events', 'USER_CREATED', { id: user.id, email: user.email });
    return user;
};

2. Controller (Interface Layer)

The Controller's sole task is to receive requests, call use cases, and return responses. Very clean!

import { Request, Response } from 'express';
import { createUserUseCase } from '@/usecases/createUser';

export const createUser = async (req: Request, res: Response) => {
    try {
        const { name, email } = req.body;
        const user = await createUserUseCase(name, email);
        res.status(201).json(user);
    } catch (err) {
        res.status(500).json({ error: 'Internal Server Error' });
    }
};

Step 6: Docker for Production Excellence

1. `Dockerfile` (Multi-stage Build)

We separate the build environment from the runtime to minimize image size and attack surface:

FROM node:22-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM node:22-alpine AS production
WORKDIR /app
ENV NODE_ENV=production
ENV NPM_CONFIG_UPDATE_NOTIFIER=false
COPY package*.json ./
RUN npm ci --only=production
COPY --from=builder /app/dist ./dist
EXPOSE 3000
CMD ["npm", "start"]

2. `docker-compose.yml` (The Full Stack)

One command to rule them all:

services:
  app:
    build: .
    ports: ["3000:3000"]
    depends_on: [db, kafka]
    environment:
      - KAFKA_BROKER=kafka:29092
      - DB_HOST=db
  db:
    image: mysql:8.0
    ports: ["3306:3306"]
    volumes: ["./flyway/sql:/docker-entrypoint-initdb.d"]
  kafka:
    image: confluentinc/cp-kafka:7.4.0
    depends_on: [zookeeper]
    ports: ["9092:9092"]

One Last Surprise... 🤫

Think this took hours to set up?

👉 The Truth Is: The entire project structure—Clean Architecture, Kafka Producers/Consumers, Flyway migrations, Docker configs, and CI/CD pipelines—was generated in under 60 seconds using an automation tool I built.

Time-to-market is everything. Stop reinventing the wheel and start shipping business value.

Want to generate a "perfect" Node.js repo like this yourself? Try my CLI tool:

npx nodejs-quickstart-structure init

Read the full document here: Nodejs Quickstart Structure

Building production-ready software shouldn't be a chore. I hope this helps you ship your next big idea faster than ever. If you found this useful, don't forget to give a Star ⭐ on GitHub! 🔥

The Hybrid Era: Why Clean Architecture needs AI-Native Scaffolding

Pau Dang — Thu, 12 Mar 2026 14:12:37 +0000

Beyond Boilerplates: Why "Classic Engineering" is the Secret Sauce for AI Success

“Engineering provides the roots; AI provides the wings. A project without roots will be blown away by the first storm of complexity.”

The 3,000 Downloads Milestone: A Reality Check

When my project, nodejs-quickstart-structure, approached the 3,000 downloads mark on NPM, I didn't just feel pride—I felt a sense of urgency.

In the age of Cursor, Claude, and Gemini, I realized that providing a "clean" structure isn't enough anymore. We are all coding with AI now, but AI has a dark secret: It is an "instant-noodle" developer. Without a strict framework, AI will generate spaghetti code at the speed of light, breaking the very architectures we spend years mastering.

That is why I released v1.15.0. It’s not just an update; it’s a manifesto for the Hybrid Era of Software Engineering.

1. The Root-and-Wing Philosophy: Amplify the Standard, Don't Mask the Flaws

90% of current AI-driven projects are built on "shifting sands." When you feed a hollow prompt into an AI, it "hallucinates" a structure from scratch.

In this ecosystem, The Roots are MVC, Clean Architecture, and a strict 70% Unit Test coverage mandate. AI is not the architect here; it is the Elite Builder executing a perfect, pre-defined blueprint.

Key Takeaway: AI can only help you soar (The Wings) if you are anchored by a solid foundation (The Roots). We use AI to amplify high standards, not to hide technical debt.

2. Solving the Developer’s Dilemma: Velocity vs. Integrity

Modern developers are caught in a crossfire:

Management demands Velocity: "We need this feature by tonight."
Senior Architects demand Integrity: "The code must be clean, tested, and scalable."

Usually, you have to sacrifice one for the other. Our tool introduces "Guardrailed Velocity." By integrating AI-native configurations directly into the scaffolding, you get the speed of AI-assisted generation without sacrificing the soul of the project. The AI is "caged" within your standards.

Key Takeaway: We don’t just sell a boilerplate; we provide an AI-Ready Infrastructure. It is the only way to move fast without breaking things.

3. The Great Shift: From Scaffolding to Knowledge Encoding

Traditional scaffolders are just "photocopiers"—they spit out files and leave the developer to figure out the rest. We are moving from a Tool to an Ecosystem.

With the new .cursorrules and prompts/ directory (Agent Skills), we are performing Knowledge Encoding. You aren't just generating code; you are transferring your "Architectural DNA" into the AI’s brain.

.cursorrules: Acts as a permanent guardrail. The moment you open the project, the AI knows it must use ApiError and must write tests before services.
prompts/: Pre-defined "Agent Skills" that teach the AI exactly how to evolve the codebase without human hand-holding.

Key Takeaway: You are no longer just a coder; you are a Technical Orchestrator managing an AI swarm that has been perfectly trained on your specific codebase.

4. Building the Future Together

This journey from 1 to 3,000 downloads has taught me that the community doesn't just want more code—they want better ways to build. The Hybrid Era is here. It’s time to stop treating AI like a magic wand and start treating it like a Senior Partner that needs a clear, disciplined environment to succeed.

🚀 Try v1.15.1 now: npx nodejs-quickstart-structure

⭐ Join the movement on GitHub: [https://github.com/paudang/nodejs-quickstart-structure]

If you believe that the future of coding is the perfect marriage between Classic Engineering and AI-Native speed, I’d love to have your support.

How I Used an AI Agent to "Enforce" 70% Unit Test Coverage for 3,000 Users

Pau Dang — Wed, 11 Mar 2026 01:09:07 +0000

Hello everyone, I'm back!

Following my previous post on Zero-Downtime, my CLI tool nodejs-quickstart-structure has fortunately reached nearly 3,000 downloads. This milestone is both a joy and a pressure. The pressure lies in: How do I ensure that every project initialized from the tool is truly "Production-Ready"?

The answer doesn't lie in boilerplate code, but in Unit Testing. However, instead of just manually writing tests, I tried a more "trendy" approach: Building an AI Testing Agent to establish a Quality Gate for the entire project.

1. The Pain: "I'll Write Tests Later!" ... and the Reality

We all know Unit Testing is important. But in reality, when starting a new Node.js project:

Setting up Jest and ts-jest takes time.
Mocking Mongoose, Redis, and Express Request/Response is incredibly cumbersome.
Defining a tests/ folder structure that follows Clean Architecture is a hassle.

The result? We often tell ourselves: "Let's finish the logic first, then write tests later." And that "later" usually never comes. With 3,000 users relying on my tool, I couldn't let that happen.

2. The Solution: From AI Chatbot to AI Agent (Thinking in Loops)

Instead of just using AI to "ghostwrite" a few isolated test snippets, I built a Testing Agent Skill. Unlike a simple Chatbot, this Agent operates based on a Standard Operating Procedure (SOP):

The "Skill" Framework I established for the Agent:

Role: Senior QA & Testing Architect.
Standards: Utilize the AAA Pattern (Arrange - Act - Assert), 100% Isolation (never touch the real database).
Goal: Enforce a minimum coverage threshold of 70% Lines and 70% Functions.
Protocol: Must list Scenarios (Happy/Sad/Edge cases) before writing any code.

Git file: https://github.com/paudang/nodejs-quickstart-structure/blob/main/docs/testing-agent-standards.md

3. Implementation: Turning "Senior Mindset" into Automation

I integrated this logic directly into the project through a system of Templates (.ejs) and smart path handling.

The Automatic "Quality Gate" Mechanism:

When you use the command to initialize a project:

The tool automatically scans the src/ structure (Controllers, Services, Resolvers).
The AI Agent suggests standard Mocks for Mongoose (e.g., admin.ping failure) or GraphQL Context.
All .spec.ts files are generated into the tests/ directory with the corresponding structure.

Fun fact: During development, I encountered several "missing file extension" bugs due to manual path handling. I eventually refactored everything to use path.parse() so the tool can run flawlessly across Windows, MacOS, and Linux.

4. Results: Numbers Don't Lie

Here is the Coverage report after I verified all scenarios for version v1.14.0:

Category	Coverage	Requirement	Status
Lines	78.5%	> 70%	✅ PASS
Functions	82.1%	> 70%	✅ PASS
Branches	70.2%	> 65%	✅ PASS

Now, every user downloading nodejs-quickstart-structure has no excuse to skip testing. The hardest "skeleton" has already been prepared by the AI.

5. Lessons Learned: AI Doesn't Replace Us, It Levels Us Up

Using an AI Agent to build Testing standards taught me one thing: AI is most powerful when placed within a standard Workflow.

If you're struggling to set up a professional Node.js project, give my tool a try:

npx nodejs-quickstart-structure init

And remember: "Quality is not an act, it is a habit."

Conclusion

My next goal is the 5,000 downloads milestone. If you find this approach useful, please leave a Star on GitHub or comment with your thoughts on how you handle Unit Testing!

GitHub Repo: https://github.com/paudang/nodejs-quickstart-structure

Thanks for reading! I hope this sharing gives you a little more motivation in your "struggle" with Unit Testing.

Stop Holding Your Breath Every Time You Deploy: Zero-Downtime Secrets for Node.js Developers

Pau Dang — Thu, 05 Mar 2026 07:37:11 +0000

Hello dev community,

Have you ever experienced that "heart-pounding" feeling when typing the deploy command to Production?

The terminal screen scrolls, and you hold your breath, praying that no 502 Bad Gateway errors appear, or worse, that your first users start complaining about sudden connection drops. In the world of modern microservices, "turning it off and on again" (Cold Restart) is a luxury we should no longer accept.

Today, I’ll share three heavy-hitting "weapons" to turn your deployment process into a smooth, Zero-Downtime experience.

1. Stability: The "Fail-fast" Mindset with Env Validation

undefined errors due to missing environment variables are the #1 enemy of stability. Has your app ever run for 5 minutes only to crash because of a missing API Key in the .env file?

Apply Defensive Programming: Validate all environment variables as soon as the application starts (Bootstrap phase). Use Zod for schema validation and type safety. If the configuration isn't clean, the app should never even "start its engine."

// src/config/env.ts
import { z } from 'zod';
import dotenv from 'dotenv';

dotenv.config();

const envSchema = z.object({
  NODE_ENV: z.enum(['development', 'test', 'production']).default('development'),
  PORT: z.string().transform(Number).default('3000'),
  DATABASE_URL: z.string().url(),
  JWT_SECRET: z.string().min(32),
});

const parsed = envSchema.safeParse(process.env);

if (!parsed.success) {
  console.error('❌ Invalid environment variables:', parsed.error.format());
  process.exit(1); // Stop the error at the source
}

export const env = parsed.data;

2. Reliability: Beyond "Up and Running" — Ensuring Application Health

Many developers stop at just checking if the process is running. In reality, a Node.js application can be "alive" but unable to serve requests (e.g., lost Database connection).

Advanced Health Checks (/health)

Your monitoring system (like a Load Balancer) needs to know the exact status of dependent resources via Deep Database Pings:

router.get('/health', async (req, res) => {
  const dbStatus = await database.ping(); 

  const healthData = {
    uptime: process.uptime(),
    status: dbStatus ? 'UP' : 'DOWN',
    timestamp: new Date().toISOString(),
  };

  res.status(dbStatus ? 200 : 503).json(healthData);
});

Graceful Shutdown: The Art of a "Smooth Exit"

When you update code, don't just "pull the plug." Implement a workflow to handle remaining requests:

process.on('SIGTERM', () => {
  logger.info('SIGTERM received. Starting graceful shutdown...');

  server.close(async () => {
    await database.disconnect(); 
    logger.info('Process terminated safely.');
    process.exit(0);
  });
});

3. Deployment: Mastering the Game with PM2 Ecosystem

On EC2 or Virtual Machines, PM2 is your most powerful operational assistant. To achieve Zero-Downtime, we need to leverage Cluster Mode and the Reload mechanism:

// ecosystem.config.js
module.exports = {
  apps: [{
    name: "nodejs-app",
    script: "./dist/server.js",
    instances: "max",       // Utilize all CPU cores
    exec_mode: "cluster",   
    wait_ready: true,       // Wait for the app to be ready before routing traffic
    listen_timeout: 3000,
    kill_timeout: 5000,     // Wait time for Graceful Shutdown
  }]
}

4. The Optimal Solution: nodejs-quickstart-structure

Setting up all these standards manually often takes at least 2 days for every new project. That’s why the nodejs-quickstart-structure project was born.

In less than a month since its launch, this tool has reached over 2,000+ downloads on NPM, proving the massive demand for a standard Node.js structure.

Initialize a Production-ready project instantly:

npx nodejs-quickstart-structure init

Why should you try it today?

✅ Architecture: Choose between MVC or Clean Architecture (TypeScript).

✅ Zero-Downtime: Built-in Graceful Shutdown & PM2 Ecosystem.

✅ Stability: Strict environment variable control with Zod.

✅ Security: Helmet, Rate Limiting, and CORS included out of the box.

✅ Observability: Professional logging system and automated Swagger Docs.

👉 Explore the source code and support the project at: https://github.com/paudang/nodejs-quickstart-structure

Thanks for reading! If you find this article and tool helpful, please consider leaving a Star for the project!

Beyond Try-Catch: Architecting a Production-Ready Error Handling System in Node.js

Pau Dang — Mon, 02 Mar 2026 13:03:50 +0000

Hello DEV community,

In Node.js application development, try-catch is just the tip of the iceberg. Catching errors is easy, but managing errors so the system remains stable and highly traceable is the true challenge for a software architect.

A standard system not only answers the question "What error just occurred?", but must also answer: "Where did this error occur, in what context, and how much should the Client know about it?"

1. The Essence of a Centralized Error Handling System

To achieve consistency, we need to gather all error handling into a single centralized point. A robust Centralized Error Handling system is built on 3 core pillars:

Categorization: Clearly separate operational errors (invalid input data, expired sessions) from programming/system errors (database crashes, code logic errors).
Contextual Logging: Ensure every thrown error carries the "trace" of the function or module where it originated, making debugging in Production multiple times faster.
Data Sanitization: Stop the risk of leaking sensitive system information to the outside world through raw error messages from the Database or Server.

2. The 3-Layer Strategy: Building a Solid "Defense Line"

Instead of letting errors "drift" freely, we force them through a strict control process across 3 layers:

Layer 1: Error Identification with Custom Error Classes

Don't throw a lifeless String. Build Classes that inherit from the Error object (e.g., AppError). Here, we attach a statusCode (for standard REST responses) and an isOperational flag (to know if this error is anticipated).

class AppError extends Error {
  constructor(message, statusCode) {
    super(message);
    this.statusCode = statusCode;
    this.isOperational = true; // Mark this as a predictable operational error
    Error.captureStackTrace(this, this.constructor);
  }
}

Layer 2: Error Mapping & Contextual Catching (The Checkpoint at the Controller)

At the Controller or Repository level, using try-catch is not merely to catch errors, but to enrich the error data.

Contextual Logging: Actively log exactly which function is experiencing the issue.
Error Mapping: Transform raw technical errors from the Database into user-friendly operational errors for the end user.

// Practical example down at the Controller
try {
  const user = await userService.createUser(req.body);
  res.status(201).json(user);
} catch (error) {
  // Actively log specific context for the failing function
  logger.error(`Error in [createUser] controller: ${error.message}`);
  next(error); // Push the error to the centralized Error Middleware
}

Layer 3: Global Error Middleware - The Final Control Station

This is the single gathering point authorized to send the Response back to the Client. Here, we perform:

Environment Filtering: In the Development environment, return full error details and stack traces. In the Production environment, return only safe messages.
Professional Logging: Deeply integrate with professional Logger libraries like Winston to categorize errors by priority level (Error, Warn, Info).

const errorMiddleware = (err, req, res, next) => {
  let error = err;

  if (!(error instanceof ApiError)) {
    const statusCode = err.statusCode || 500;
    const message = error.message || 'Internal Server Error';
    error = new ApiError(statusCode, message, false, err.stack);
  }

  const { statusCode, message } = error;

  if (statusCode === 500) {
    logger.error(`${statusCode} - ${message} - ${req.originalUrl} - ${req.method} - ${req.ip}`);
    logger.error(error.stack || 'No stack trace');
  }

  res.status(statusCode).json({
    statusCode,
    message,
    ...(process.env.NODE_ENV === 'development' && { stack: error.stack }),
  });
};

3. Automating Standards with nodejs-quickstart-structure

Manually setting up the entire process above for every new project is a waste of time and prone to errors. The project nodejs-quickstart-structure was born to help you enforce these architectural standards automatically and consistently.

Why should you use nodejs-quickstart-structure?
The project provides absolute flexibility by letting you choose the architecture you want, not a mandatory Framework architecture:

Built-in Centralized Error Handling System: All Middlewares, Error Classes, and Winston configurations are pre-set according to practical operational standards.
Architecture Customization: You can choose MVC for lean projects or Clean Architecture to protect the integrity of Business Logic.
Production-Ready: Built-in Winston Logging, basic security configurations (Helmet, CORS), and optimized Docker Multi-stage image size.

Initialize a standard project with just one command:

npx nodejs-quickstart-structure init