Forem: John Doe

The Hidden Security Risk in Your NestJS App: A Validation Story

John Doe — Fri, 18 Jul 2025 17:17:53 +0000

After spending years consulting on legacy NestJS projects, I've seen a pattern that keeps me up at night. It's not the complex business logic or the tangled dependencies that worry me most - it's something far simpler: validation.

Unlike a broken feature that screams for attention, poor validation quietly accumulates until one day, you're staring at a SQL injection attack or a crashed server, wondering how it happened.

Why Your First Line of Defense Matters More Than You Think

Picture your typical NestJS application. Data flows like water through pipes:

HTTP request → Controller → App Service → Database

When you validate at the entry point—right at those Request DTOs—you're essentially installing a filter at the source. Skip this step, and contaminated data flows through your entire system.

The Validation Mistakes I See Over and Over

Let me share what I've discovered lurking in production codebases.

1. The "It's a String, Ship It!" Syndrome

This is the classic:

export class CreateUserDto {
  @IsString()
  name: string;

  @IsString() 
  description: string;
}

Sure, you've confirmed it's a string. But what about:

That user who submits a 10MB "name"?
The clever hacker testing '; DROP TABLE users; --?
Someone pasting <script>alert('gotcha')</script> in the description?

2. The Wild West of Numbers

I've lost count of how many times I've seen:

@IsNumber()
quantity: number;  // What could go wrong?

Well, let me tell you what goes wrong:

Negative quantities in shopping carts (free money, anyone?)
Page numbers like 999999999 that bring your database to its knees
Array generation based on user input that eats all your RAM

3. The String That Ate Production

This one keeps me awake:

@IsString()
token: string;  // Living dangerously

Without content validation, you're inviting:

SQL injection attempts
JavaScript payloads trying to access your environment variables
Path traversal attacks reaching for /etc/passwd
XSS attacks waiting to pounce on your users

How We Got Here: A Brief History

The class-validator Revolution (2020-Present)

NestJS and class-validator changed the game, but here's the catch: most developers stop at @IsString() and call it a day. The tools are there, but using them properly requires knowledge that's often buried in documentation or learned through painful experience.

The AI Era (2023-Present)

Now we have Cursor, Claude, and other AI assistants. They're game-changers, but they're not magic:

They analyze one file at a time, missing the big picture
The output quality depends entirely on how you prompt them

What Good Validation Actually Looks Like

Let me show you the difference AI-assisted development can make.

The Old Way:

@IsString()
username: string;

@IsString()
@MaxLength(1000)
content: string;

The More Secure Way:

// User input fields
@IsString()
@MaxLength(255)
@Matches(/^[a-zA-Z0-9\s\-_]+$/, {
  message: 'Only alphanumeric characters, spaces, hyphens, and underscores allowed'
})
@Transform(({ value }) => value?.trim())
username: string;

// Fields that might contain HTML
@IsString()
@MaxLength(10000)
@Transform(({ value }) => sanitizeHtml(value, { allowedTags: [] }))
content: string;

The difference? The second approach actually thinks about what could go wrong.

The True Cost of Cutting Corners

Here's what poor validation really costs you:

Security breaches - Not if, but when. SQL injection and XSS are still alive and well in 2025.

Data corruption - Once bad data enters your database, good luck cleaning it up.

Performance nightmares - innocent-looking unbounded query? It's a ticking time bomb.

Business logic failures - Negative prices, impossible dates, calculations that make no sense.

Developer suffering - Hours spent debugging issues that proper validation would have prevented.

Scaling the Solution

For large legacy projects, updating validation is really hard. That's why I ended up building class-validator-security-enhancer, an AI-powered tool that can analyze and fix validation across entire codebases. But whether you use automation or not, the principle remains the same: treat validation as seriously as you treat your core business logic.

The Bottom Line

Here's what I want you to remember:

Validation is your moat - It stops attacks before they can do damage
Type checking is just the beginning - Real security comes from content validation
Legacy code needs love too - Systematic approaches beat ad-hoc fixes
AI is a powerful ally - But it needs your expertise to guide it
Consistency is king - One weak link can compromise everything

We're living in an era where AI can help us write secure validation code in seconds. The tools are there, the patterns are established, and the knowledge is available. If your DTOs still look like they're from 2019, it's time for an upgrade.

How to easily send emails in NestJS?

John Doe — Sun, 05 Jan 2025 21:12:46 +0000

Sending emails is a critical functionality in many applications, whether for user notifications, transactional updates, or marketing purposes. However, implementing email solutions can sometimes be cumbersome, as you have to integrate mailer with template language, check for dependencies...

But!

With the @nestixis/nestjs-mailer package, you can simplify this process while ensuring flexibility and reliability.

This package leverages the power of React and Nodemailer, making it a modern and developer-friendly tool for building dynamic email templates and sending emails effortlessly.

Let's dive into how you can set it up and use it :)

Installing the Package

To get started, you need to install the nestjs-mailer package in your NestJS application. This package is available via npm, making installation quick and straightforward. Run the following command in your terminal:

npm install @nestixis/nestjs-mailer

Configuring the Module

Once the package is installed, the next step is to configure the MailerSdkModule in your application.

Configuration is straightforward, and for testing purposes, you can use a tool like Mailcatch to capture and preview emails without sending them to real users. Here's an example of how to set it up:

import { MailerSdkModule } from '@nestixis/nestjs-mailer';
import { Module } from '@nestjs/common';
import { AppController } from './app.controller';
import { AppService } from './app.service';

@Module({
  imports: [
    MailerSdkModule.register({
      auth: {
        user: 'username',
        password: 'password',
        host: 'sandbox-smtp.mailcatch.app',
        port: 2525,
        ssl: false,
      },
      from: 'your-email@example.com',
    }),
  ],
  controllers: [AppController],
  providers: [AppService],
})
export class AppModule {}

Creating an Email Template

To make your emails visually appealing and more dynamic, you can combine templates with React, and package@react-email/components, allows you to design such email templates.

But before that, you should call the file invite-admin-with-account-template.tsx and set

"jsx": "react"

in your tsconfig.json

Here's an example of a template for inviting a new admin user:

import {
  Body,
  Container,
  Head,
  Html,
  Img,
  Link,
  Section,
  Text,
} from '@react-email/components';
import * as React from 'react';

export default function InviteAdminWithAccountTemplate({
  translation,
  language,
  invitationHref,
  passwordHref,
  logoUrl,
}) {
  return (
    <Html lang={language}>
      <Head>
        <style>{/* Your custom styles here */}</style>
      </Head>
      <Body style={{ fontFamily: 'Arial, sans-serif' }}>
        <Section>
          <Container>
            {logoUrl ? (
              <Img src={logoUrl} alt="Logo" />
            ) : (
              <Text>{translation.titleInside}</Text>
            )}
            <Text>{translation.contentPart1}</Text>
            <Text>{translation.contentPart2}</Text>
            <Text>
              {translation.contentPart3.subpart1}
              <Link href={invitationHref}>
                {translation.contentPart3.subpart2}
              </Link>
              {translation.contentPart3.subpart3}
            </Text>
            <Text>
              {translation.contentPart4.subpart1}
              <Link href={passwordHref}>
                {translation.contentPart4.subpart2}
              </Link>
            </Text>
          </Container>
        </Section>
      </Body>
    </Html>
  );
}

Injecting the Email Sender

After creating your email template, the next step is to send the email. To do this, you inject the email sender into your service.

import {
  EmailSenderInterface,
  MAILER_SDK_CLIENT,
} from '@nestixis/nestjs-mailer';
import { Inject, Injectable } from '@nestjs/common';
import InviteAdminWithAccountTemplate from './invite-admin-with-account-template';

@Injectable()
export class AppService {
  constructor(
    @Inject(MAILER_SDK_CLIENT)
    private readonly emailSender: EmailSenderInterface,
  ) {}

  async send(): Promise<void> {
    const translations = {
      titleInside: { subpart1: 'Welcome', subpart2: ' to the platform!' },
      contentPart1: 'Hello',
      contentPart2: 'Your admin account has been created.',
      contentPart3: {
        subpart1: 'Click here to activate your account: ',
        subpart2: 'Activate',
        subpart3: '.',
      },
      contentPart4: {
        subpart1: 'To set your password, click here: ',
        subpart2: 'Set password',
      },
    };

    const emailContent = await InviteAdminWithAccountTemplate({
      translation: translations,
      language: 'en',
      invitationHref: 'xxx',
      passwordHref: 'xxx',
      logoUrl: 'logo.png',
    });

    await this.emailSender.sendEmail(
      'test@test.com',
      'Admin Invitation',
      emailContent,
    );
  }
}

Done!

That's it! You've successfully integrated nestjs-mailer into your application.

For more details and advanced features, check out the NestJS mailer GitHub repository.

Advantages of search databases

John Doe — Sat, 04 Jan 2025 19:28:30 +0000

In the past, I often noticed a common approach where developers (including myself of course) used the same API for both reads and writes on every case. Even more so, we frequently relied on the same data source, such as MySQL/PostgreSQL, to handle both operations.

This means writing to the same columns and reading from them, often leading to struggles with optimizing indexes on fields that were heavily queried.

So for instance, we’d find ourselves frequently tweaking indexes to accommodate new filters or improve query performance, and fields used with operators like LIKE posed particular challenges due to their impact on performance.

These changes often lead to further adjustments to the backend, including modifying APIs to expose the updated functionality, measured times because of additional JOINs and so on...

To address the challenge about adding new filters and stuff in the API, there were attempts to optimize the process using tools and standards like Apicalypse and, of course, GraphQL.

These solutions aimed to streamline API query generation and reduce the manual effort required to implement new filters and functionalities, offering a more dynamic approach to handling data access, but, they had high learning curve.

With the rise of CQRS (Command Query Responsibility Segregation), a new approach began to emerge. This mindset encouraged the use of separate sources for writes and reads. Writes could emit events, and reads could build views from those events in dedicated places. Even if the reads and writes were managed within the same database (but different tables), this separation brought significant benefits, and, of course was able to get rid off of second challenge - JOINs and search queries on domain models, as read models are commonly in a form of denormalized JSONs.

However, this raised another problem. With reads, we had to scale writes, meaning the only reason we had to scale instances of our application from X to Y was because of reads. This issue could be partially mitigated with caching, and in the world of microservices, we could have dedicated microservices for reads.

But...

Still, this was not an ideal solution for other architectural styles like modular monoliths, where such separation might not align well with the system’s design philosophy. Another thing was, when API was down, the whole product was down, and keeping in mind, that most of the products are relying on more reads than writes, it could unnecessary impact the business (Aparat from down API of course ;) )

So, what if we could ask those "views," also known as read models, directly without involving the API and handling loads? This is where solutions like Meilisearch, AppSearch and others come into play, leveraging a pattern called the "Valet Key." By using this pattern, frontends can access read-optimized models directly, reducing the dependency on backend APIs. Of course, frontend still has to "ask" API for "Valet key", but frontend can cache keys, so even when API is down, frontend can still communicate and display content.

With this approach, we can focus on the read database and not worry about handling the traffic for reads in our API. The "Valet Key" provided to the frontend via our API is secured in a way that the frontend cannot alter it. It includes predefined filters and indexes.

If the frontend requires additional capabilities, it can request them through the API, where API can validate whether to allow them. It's still less calls.

Some pros I can see are:

Reduced API Load: Offloads the read traffic from the API, allowing it to focus on core operations.
Scalability: Read databases or search services are better optimized to handle high traffic, reducing the need to scale the application backend.
Flexibility: SaaS or self-hosted options allow teams to choose the best fit for their infrastructure.
Security: Predefined filters and indexes ensure the frontend can only access allowed data, minimizing risks. Keys can be invalidate by the API.
Developer Efficiency: Reduces the need for constant API updates for new filters or search capabilities.
Improved Performance: Direct access to read-optimized models provides faster query responses for users.

But, there are always cons cons:

Eventual Consistency: Data may appear after some time due to the nature of eventual consistency in read models.
Additional Maintenance: Introduces an extra component that requires monitoring and management.
Schema Complexity: Schemas must be stored in code or a common place, as different teams from different contexts may need to populate the same document (e.g., employee with email, but also with available credits and coupons). While not directly tied to this pattern, it adds complexity.
Cost of SaaS version or self hosted maintenance

So, this approach is not a silver bullet and introduces its own set of challenges, but if you're okay with a cons, then a small change on the frontend likely won’t require involving the backend team, streamlining the development process and improving overall agility, and of course scalability should be easier.

How to solve dual write problem in NestJS?

John Doe — Sun, 15 Sep 2024 11:39:12 +0000

There are only two hard things in Computer Science: cache invalidation and naming things.
-- Phil Karlton

Well, let's add another one to the list: consistency. In distributed systems or modular monoliths, maintaining data consistency across multiple services or modules can be quite a challenge.

Let's imagine we have an e-commerce application with two services: OrderService and PaymentService. The OrderService is responsible for confirming orders, while the PaymentService handles the payment processing. Here's a simplified example in NestJS:

// order.service.ts
@Injectable()
export class OrderService {
  constructor(
    private readonly eventEmitter: EventEmitter2,
    private readonly orderRepository: OrderRepository,
  ) {}

  async confirmOrder(orderId: string): Promise<void> {
    const order = await this.orderRepository.findOne(orderId);
    order.status = 'CONFIRMED'
    await this.orderRepository.save(order);
    this.eventEmitter.emit('order.confirmed', { orderId });
  }
}

// payment.service.ts
@Injectable()
export class PaymentService {
  constructor(
    @InjectRepository(Payment)
    private readonly paymentRepository: Repository<Payment>,
    private readonly eventEmitter: EventEmitter2,
  ) {
    this.eventEmitter.on('order.confirmed', this.handleOrderConfirmed.bind(this));
  }

 // Listener for order.confirmed
  async handleOrderConfirmed(event: { orderId: string }): Promise<void> {
      // Start payment process
      const payment = new Payment();
      payment.orderId = event.orderId;
      payment.status = 'PROCESSING';
      await this.paymentRepository.save(payment);
      // Further payment logic...
  }
}

In this scenario, multiple things could go wrong, leading to an inconsistent system:

Order could be saved, but event could not be emitted: The OrderService confirms an order, but due to a crash or code issue, the event is not emitted.
Event emitted, but Listener did not catch: The OrderService emits the event, but the PaymentService fails to catch it because of a crash or a network issue.
Listener Catches Event, But Errors Occur: The PaymentService catches the event but throws an unexpected and unhandled error during payment processing.

These issues can lead to a broken process flow and an inconsistent system state, which will may require manual debugging and pulled hair out of the head.

This is where the dual write problem appears.

But, fortunately, if there is a problem, there are solutions. Let's take at possible options:

Two-Phase Commit (2PC)

The Two-Phase Commit protocol ensures that all participating services agree to either commit or roll back a transaction, maintaining consistency across the system. The process is divided into two phases: the prepare phase and the commit phase.

Prepare Phase: The coordinator asks all participants if they can commit the transaction. Each participant performs the necessary operations and responds with a vote (yes or no).
Commit Phase: If all participants vote yes, the coordinator instructs them to commit the transaction. If any participant votes no, the coordinator instructs them to roll back.

Cons of Two-Phase Commit (2PC):

Complexity: Implementing 2PC requires significant changes to your system and additional coordination logic.
Performance Overhead: The synchronization between services can introduce latency, especially in high-throughput systems.
Single Point of Failure: The transaction coordinator can become a single point of failure, affecting the entire system's reliability.
Scalability Issues: As the number of participants increases, the protocol can become less efficient and harder to manage.

Debezium

Debezium is a distributed platform that captures row-level changes in your databases so that your applications can see and respond to those changes. It provides change data capture (CDC) functionality, which can be used to ensure consistency.

By integrating Debezium with Kafka, you can ensure that any change to your database is captured and processed reliably, providing an alternative approach to solving the dual write problem.

Here’s a brief overview of how Debezium can be integrated:

Setup Debezium: Configure Debezium connectors to monitor your database tables.
Capture Changes: Debezium captures row-level changes and publishes them to Kafka topics.
Process Events: Consume these Kafka topics in your application to process events consistently.

Cons of Debezium:

Operational Complexity: Setting up and managing Debezium, Kafka, and the necessary connectors can be complex and requires additional infrastructure.
Latency: While Debezium can capture changes in near real-time, there might still be slight delays between the database update and the event being processed.
Resource Consumption: Running Debezium and Kafka can be resource-intensive, impacting overall system performance.
Error Handling: Handling errors and ensuring reliable event processing requires careful design and monitoring to avoid data loss or inconsistencies.

Inbox and Outbox patterns

The Outbox and Inbox patterns work together to ensure that events are consistently published and processed, even in the face of failures. Here's how they can help:

Outbox Pattern: This pattern ensures atomicity by saving events in a local database table (the outbox) within the same transaction as the business operation. A separate process (Or the same at first attempt) then reads these events from the outbox and publishes them. This decouples the business logic from event publishing, ensuring that events are only published if the transaction succeeds.
Inbox Pattern: The inbox pattern involves recording received events in an inbox table to ensure idempotent processing. This prevents the same event from being processed multiple times and maintains consistency across services, also giving the ability to retry delivery if something went wrong.

When combined, these patterns provide robust solutions to the dual write problem:

Atomicity: By recording events in the outbox as part of the same transaction that modifies the business data, you ensure that either both the data change and the event are committed or neither are.
At least one delivery: You may rest assured, because your listeners will be invoked with events at least once
Decoupling: Services remain decoupled, as the producer service does not directly publish events. Instead, a separate outbox processor handles event publishing.

Cons of Outbox and Inbox patterns

Operational Overhead: Implementing and maintaining outbox and inbox tables and the corresponding processors can add complexity to your system.
Latency: There can be a delay between the business operation and the event processing, depending on how frequently the inbox processor runs.
Resource Consumption: Running additional background jobs for inbox processing can consume extra resources and may impact system performance.

While the outbox and inbox patterns do introduce some additional complexity and potential latency, these cons are generally less significant than those associated with the Two-Phase Commit (2PC) protocol or Debezium, that's why, we will stick with combination of Inbox and Outbox patterns to solve dual write problem.

Implementation

To mitigate some of the operational overhead and latency introduced by the traditional outbox and inbox patterns, you can use the newly created NestJS Inbox Outbox Package from Nestixis. This package aims to streamline the implementation and operation of these patterns, solving some problems differently from the standard approach.

Disclaimer: Nestixis is an organization that I've created recently. The goal is to provide useful packages and give back something useful to community

Immediate Event Delivery: The package mimics EventEmitter2, delivering events to all listeners immediately at first try, reducing latency.
Retry Mechanism: If one or more listeners fail, they will be retried based on a given interval.
Retry Conditions: Listeners will be retried if either the expireAt time has not yet passed or until the listener stops throwing exceptions.
Pessimistic Execution for Scalability: The drivers for MikroORM and TypeORM perform polling and execute events in a pessimistic way, enabling better scaling across multiple services.
Extendability: Due to easy interface to implement interface, it should not be a problem to create custom drivers for e.g. drizzle or prisma.

Inbox pattern used in the package

Outbox pattern used in the package

Prerequisites

Docker
Node.js and npm
NestJS CLI

Step 1: Set Up PostgreSQL Database

First, we'll use Docker to set up our PostgreSQL database:

docker run -d \
  --name postgres-container \
  -e POSTGRES_DB=inbox_outbox \
  -e POSTGRES_USER=user \
  -e POSTGRES_PASSWORD=user \
  -p 5432:5432 \
  postgres

This command creates a PostgreSQL container named postgres-container with a database called inbox_outbox.

Step 2: Initialize NestJS Project
Create a new NestJS project:

nest new nestjs-inbox-outbox-example
cd nestjs-inbox-outbox-example

Step 3: Install Required Packages

npm i @nestjs/typeorm typeorm pg @nestixis/nestjs-inbox-outbox @nestixis/nestjs-inbox-outbox-typeorm-driver

Step 4: Create Event and Listener

Create a new file src/new-cat.event.ts:

export class NewCatEvent implements InboxOutboxEvent {
  public readonly name = NewCatEvent.name;

  constructor(
    public readonly catName: string,
    public readonly isFluffy: boolean,
  ) {}
}

Create a listener in src/new-cat.listener.ts:

import { IListener, Listener } from '@nestixis/nestjs-inbox-outbox';
import { NewCatEvent } from './new-cat.event';

@Listener(NewCatEvent.name)
export class NewCatListener implements IListener<NewCatEvent> {
  getName(): string {
    return NewCatListener.name;
  }

  async handle(event: NewCatEvent): Promise<void> {
    console.log(`A new and cute cat appeared with name ${event.catName}`);
  }
}

Step 5: Create Cat Model
Create src/cat.model.ts:

import { Column, Entity, PrimaryGeneratedColumn } from 'typeorm';

@Entity()
export class Cat {
  @PrimaryGeneratedColumn()
  id: number;

  @Column()
  name: string;
}

Step 6: Set Up Database Table
Connect to your PostgreSQL database and execute:

CREATE TABLE "cat" (
    "id" SERIAL PRIMARY KEY,
    "name" VARCHAR NOT NULL
);

Step 7: Configure AppModule
Update src/app.module.ts:

import { InboxOutboxModule } from '@nestixis/nestjs-inbox-outbox';
import {
  InboxOutboxTransportEventMigrations,
  TypeORMDatabaseDriverFactory,
  TypeOrmInboxOutboxTransportEvent,
} from '@nestixis/nestjs-inbox-outbox-typeorm-driver';
import { Module } from '@nestjs/common';
import { TypeOrmModule } from '@nestjs/typeorm';
import { DataSource } from 'typeorm';
import { Cat } from './cat.model';
import { NewCatEvent } from './new-cat.event';
import { NewCatListener } from './new-cat.listener';

@Module({
  imports: [
    TypeOrmModule.forRoot({
      type: 'postgres',
      host: 'localhost',
      port: 5432,
      username: 'user',
      password: 'user',
      database: 'inbox_outbox',
      entities: [TypeOrmInboxOutboxTransportEvent, Cat],
      migrations: [...InboxOutboxTransportEventMigrations],
      logging: true,
      migrationsRun: true,
    }),
    InboxOutboxModule.registerAsync({
      isGlobal: true,
      imports: [TypeOrmModule.forFeature([TypeOrmInboxOutboxTransportEvent])],
      useFactory: (dataSource: DataSource) => {
        const driverFactory = new TypeORMDatabaseDriverFactory(dataSource);
        return {
          driverFactory: driverFactory,
          events: [
            {
              name: NewCatEvent.name,
              listeners: {
                expiresAtTTL: 1000 * 60 * 60 * 24,
                maxExecutionTimeTTL: 5000
                readyToRetryAfterTTL: 10000,
              },
            },
          ],
          retryEveryMilliseconds: 15000,
          maxInboxOutboxTransportEventPerRetry: 10,
        };
      },
      inject: [DataSource],
    }),
  ],
  providers: [NewCatListener],
})
export class AppModule {}

Step 8: Update main.ts to Emit Events
Update src/main.ts:

import {
  TransactionalEventEmitter,
  TransactionalEventEmitterOperations,
} from '@nestixis/nestjs-inbox-outbox';
import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';
import { Cat } from './cat.model';
import { NewCatEvent } from './new-cat.event';

async function bootstrap() {
  const app = await NestFactory.create(AppModule);
  await app.init();

  const transactionalEventEmitter = app.get<TransactionalEventEmitter>(
    TransactionalEventEmitter,
  );

  const cat = new Cat();
  cat.name = 'Mr. Whiskers';

  await transactionalEventEmitter.emit(new NewCatEvent(cat.name, true), [
    {
      entity: cat,
      operation: TransactionalEventEmitterOperations.persist,
    },
  ]);

  await app.listen(3000);
}
bootstrap();

Step 9: Run the Application

Start your NestJS application:

nest start

You should see a log message indicating that a new cat has appeared.

Conclusion

Congratulations! You've now set up a NestJS application implementing the Inbox/Outbox pattern with PostgreSQL.

As you continue to develop your application, pay extra attention to the following points:

Idempotent Listeners: Ensure your event listeners are idempotent, meaning they can be executed multiple times without changing the result beyond the initial application. This is crucial for handling potential duplicate events.
Unique Names: The package checks for unique names of events and listeners at startup. Always use distinct, descriptive names for your events and listeners to avoid conflicts.
Execution Time: Consider the execution time of your listeners. If they perform long-running operations, adjust the maxExecutionTimeTTL accordingly. This prevents premature timeouts for complex tasks.
Message Expiration: Not all messages need to be repeated indefinitely. Use the expiresAtTTL parameter wisely to set appropriate expiration times for your events. This helps manage system resources and prevents processing of outdated events.
Retry Cooldown: The readyToRetryAfterTTL parameter sets a cooldown period after the first failure of a listener. Consider the nature of your operations when setting this value. For transient failures, a shorter cooldown might be appropriate, while for dependent service issues, a longer cooldown could be beneficial.

Link to working example: Github

Why you should start profiling your application on the production

John Doe — Wed, 01 Nov 2023 17:03:16 +0000

Let's start with a common situation.

You've just created a new application in node.js. The application uses some SQL database, redis for cache, Kafka as a broker, and many many other useful things.

You are ready to deploy your masterpiece on production, so you choose your host provider, set up a domain, and finally - deploy the application. Everything works fine so far.

But after a few days, you have calls and emails from your clients. The application has started to work slower and slower, you are taking a look at your infrastructure metrics and you catch memory and CPU spikes, but when you take a look at APM you see no usual spikes in requests, everything looks the same as yesterday. At this point, you are blind.

This hypothetical case sounds scary, but usually, restarting fixes the problem and you can buy some time to investigate what actually happens.

We could easily avoid this situation, partially, if we had enabled profiling on production. But what the heck is profiling? Let's get familiar with the definition

Profiling, in a general sense, refers to the process of monitoring and analyzing how a computer program utilizes CPU and memory resources. It is a crucial technique for identifying performance bottlenecks, optimizing code, and enhancing the efficiency of software applications during the development and testing phases.

So in general, we want to track how memory and processor are used by our application and which part of our applications has a problem like

Memory leaks
Processor abuse
Slow connections
I/O bottlenecks
Ineffective caching
Third-party library issues

Let's see what kind of tools can we use in production to measure and profile the application

Google Profiler

Cloud Profiler is a statistical, low-overhead profiler that continuously gathers CPU usage and memory-allocation information from your production applications. It attributes that information to the source code that generated it, helping you identify the parts of your application that are consuming the most resources, and otherwise illuminating your application's performance characteristics.

Google Cloud Profiler has been a good friend of mine when it comes to finding what function or class has been naughty.

But as with any tool, it has some flaws, like showing you... too much

Whole application heap

But, at the end of the day, it does its job by helping you find your code and 3rd library usage

(Usage of processor - wall time)

(Usage of memory - heap)

Ability to see top CPU and memory functions

Ability to compare profiles in time and by versions

It's basic, but it works. You can see what function or 3rd library is after your resources and debug them locally (For example, by using ClinicJS - https://clinicjs.org/)

Pros:

It's cheap, you can use it in as many instances as you want
Easy to install
You only need to install this as a package and include it in your main file

Cons:

UI is very laggy
Wall time is sometimes not useful without many filters
Lack of correlations with infrastructure
Capture profile every 10 minutes
Higher memory usage because it's stores temporary profiles in application memory

Link to Google Cloud Profiler https://cloud.google.com/profiler/docs/about-profiler

DataDog

DataDog is a far bigger player when it comes to APM and infrastructure monitoring as well as when it comes to profiling applications. To be honest, DataDog is my current top 1 tool for profiling in the Node.JS environment.

And there is a simple explanation for that - they understand how Node.JS works and adjust profiling to show you what you really need to see

At first, they allow you to see the timeline for wall time and heap usage, so you can spot that something is off at first glance.

Secondly, they allow you to see only your code. Moreover, you can see at what point in time the profile has been captured

What I also find quite useful, is that the UI is not laggy at all, even when you have hundreds of services on the UI

Pros:

Smart and fast UI
Capture profiler every minute
Correlations with infrastructure, logs, etc.
Adjusted to Node.JS needs, you only see what's important

Cons:

Kinda expensive, 48 USD per host + 18 USD per infra
You need to install the agent on the infrastructure and then install the package in your code in order to communicate with the agent

Lint to DataDog profiler: https://docs.datadoghq.com/profiler/

So as you can see, those tools can help you drastically reduce the time of debugging and make your application and clients happy again.

A smart way to create scalable web sockets in NestJS

John Doe — Sat, 28 Oct 2023 18:39:43 +0000

A long time ago, I found myself in a situation where I had to create a scalable system that could be capable of handling hundreds of simultaneous connections at not very high cost, and with reasonable, but not instant time of response.

My first thoughts? Let's move all create/edit/delete actions to the queue and notify users if their actions succeeded or not via WebSocket.

But back then, I hadn't much experience with WebSockets in production, so my first step was to investigate how it works with the help of tutorials, stack overflow, and other sources.

So, after some time, I got a gist of how it should work and started to prepare a code and mess around for a while with a load tests tool to simulate high traffic.

The first problem

Some questions and answers suggested calling the subscribe method on the Redis instance on the connected WebSocket client.

io.sockets.on('connection', function (sockets) {
    sockets.emit('message',{Hello: 'World!'});
    sub.subscribe('attack-map-production');

    sockets.on('disconnect', function() {
            sub.unsubscribe('attack-map-production');
    });
});

But in this way, we are creating a new connection to Redis, so the memory usage in our application and Redis connection pool are rising. (Redis allows only 10k connections to one instance)

That was a big no for me because I had to lower memory usage to a minimum.

Right now, many articles, fortunately, mention that you should not create a new Redis connection on each WebSocket client.

The second problem

After creating a big chunk of business code, when I started the part with web sockets, a question popped into my mind - how to create them in a proper and safe way?

I already had some events in the system, and some of them were ready to be additionally published via WebSockets, the rest of them were meant to stay inside the system.

My gold promise was that I wouldn't have to change code drastically and still be able to send only selected events to websocket clients.

That's why, at first, I created a Redis pub-sub module, as I thought that my events, in order to be visible in other instances, have to be transmitted via the Redis pub-sub pattern.

Don't feel overwhelmed by looking at the module below, as I will explain details later in a use case


export const REDIS_PUB_CLIENT = 'REDIS_PUB_CLIENT';
export const REDIS_SUB_CLIENT = 'REDIS_SUB_CLIENT';
export const REDIS_EVENT_PUB_SUB_REGISTER_EVENT_OPTIONS =
  'REDIS_EVENT_PUB_SUB_REGISTER_EVENT_OPTIONS';

@Module({
  providers: [
    {
      provide: REDIS_EVENT_PUB_SUB_REGISTER_EVENT_OPTIONS,
      useFactory: (options: RedisEventPubSubModuleOptions) => options,
      inject: [MODULE_OPTIONS_TOKEN],
    },
    {
      provide: REDIS_PUB_CLIENT,
      useFactory: async (options: RedisEventPubSubModuleOptions) => {
        const client = createClient({
          url: `redis://${options.host}:${options.port}`,
        });
        client.on('error', (err) => console.error('Redis Client Error', err));
        await client.connect();
        return client;
      },
      inject: [MODULE_OPTIONS_TOKEN],
    },
    {
      provide: EVENT_EMITTER_TOKEN,
      useFactory: (
        redisPubClient: RedisClientType,
        eventEmitter: EventEmitter2,
      ) => {
        return new RedisEventEmitter(redisPubClient, eventEmitter);
      },
      inject: [REDIS_PUB_CLIENT, EventEmitter2],
    },
    {
      provide: EVENT_SUBSCRIBER_TOKEN,
      useFactory: (eventEmitterSub: EventEmitter2) => {
        return new EventEmitter2EventSubscriber(eventEmitterSub);
      },
      inject: [EventEmitter2],
    },
  ],
  exports: [
    REDIS_PUB_CLIENT,
    EVENT_EMITTER_TOKEN,
    EVENT_SUBSCRIBER_TOKEN,
    REDIS_EVENT_PUB_SUB_REGISTER_EVENT_OPTIONS,
  ],
})
export class RedisEventPubSubModule extends ConfigurableModuleClass {
  static registerEvents(eventsPublishableNames: string[]): DynamicModule {
    return {
      module: class {},
      providers: [
        {
          provide: REDIS_SUB_CLIENT,
          useFactory: async (
            options: RedisEventPubSubModuleOptions,
            eventEmitter: EventEmitter2,
          ) => {
            const client = createClient({
              url: `redis://${options.host}:${options.port}`,
            });
            client.on('error', (err) =>
              console.error('Redis Client Error', err),
            );
            await client.connect();
            for (const eventPublishableName of eventsPublishableNames) {
              await client.subscribe(eventPublishableName, (message) => {
                const normalizedMessage = JSON.parse(
                  message,
                ) as PublishableEventInterface;
                delete (
                  normalizedMessage as Writeable<PublishableEventInterface>
                ).publishableEventName;
                eventEmitter.emit(eventPublishableName, normalizedMessage);
              });
            }
            return client;
          },
          inject: [REDIS_EVENT_PUB_SUB_REGISTER_EVENT_OPTIONS, EventEmitter2],
        },
      ],
    };
  }
}

This module takes care of creating/exposing a Pub Redis client and exposing an additional method - registerEvents, which is responsible for listening for given events on Redis pub-sub and re-emitting them via event emitter.

It may be a little foggy for now. Why re-emitting events? Why do we need to register for those events? What are EVENT_EMITTER_TOKEN and EVENT_SUBSCRIBER_TOKEN and why do we have to export them?

It will be more clear with real-life usage, so let's create a use case - chat messages. We want to be able to send messages via HTTP POST and receive them via WebSocket on the front end.

Let's begin

Publishing events

Here's a module for that


@Module({
  imports: [],
  controllers: [],
  providers: [],
})
export class UserChatModule {}

And an event that this module will be emitting after receiving a POST request

export class NewMessageEvent {
  constructor(public readonly message: string) {}
}

In the controller, we have to make it possible to emit events both for our system and the Redis pub queue. We will use wrapped EventEmitter2 for that


export const EVENT_EMITTER_TOKEN = 'EVENT_EMITTER_TOKEN';

export class RedisEventEmitter implements EventEmitterInterface {
  constructor(
    private redisPubClient: RedisClientType,
    private eventEmitter: EventEmitter2,
  ) {}

  async emit(eventName: string, payload: Record<any, any>): Promise<void> {
    this.eventEmitter.emit(eventName, payload);

    if (this.isPublishableEvent(payload)) {
      await this.redisPubClient.publish(
        payload.publishableEventName,
        JSON.stringify(payload),
      );
    }
  }

  private isPublishableEvent(event: any): event is PublishableEventInterface {
    return event.publishableEventName !== undefined;
  }
}

And then, we are able to use it in our controller

@Controller('messages')
export class SendMessageAction {
  constructor(
    // Previously eventEmitter2
    @Inject(EVENT_EMITTER_TOKEN)
    private readonly eventEmitter: EventEmitterInterface,
  ) {}

  @Post()
  async handle(@Body() request: SendMessageHttpRequest) {
    await this.eventEmitter.emit(
      NewMessageEvent.name,
      new NewMessageEvent(request.content),
    );
  }
}

But before that, we have to enhance our event with PublishableEventInterface in order to allow RedisEventEmitter to catch our event and emit it in the Redis pub queue.

export class NewMessageEvent implements PublishableEventInterface {
  static publishableEventName = 'events:new-message';

  publishableEventName = NewMessageEvent.publishableEventName;

  constructor(public readonly message: string) {}
}

Great, we are now sending our events like we used to, but now, if they are marked as publishable, they will land in the Redis pub queue.

But now, we need to make it possible to receive those events on WebSocket, right?

Receiving events

So, let's take a look at our user chat module

@Module({
  imports: [
    RedisEventPubSubModule.registerEvents([
      NewMessageEvent.publishableEventName,
    ]),
  ],
  controllers: [SendMessageAction],
  providers: [],
})
export class UserChatModule {}

As you can see, we used the method mentioned earlier - registerEvents.

Thanks to that method, we told RedisEventPubSubModule that it should listen for our NewMessageEvent event in the Redis pub-sub queue on the publishableEventName attribute.

So, if any NewMessageEvent event occurs, then it will be re-emitted as a normal NewMessageEvent event, but under the publishableEventName attribute.

It's worth mentioning, that it will work on 1 instance or 1,000 instances. So even if we scale to a high number of instances, each of them will receive this and re-emit this event inside the system.

So, now we have abilities to emit events and listen for them. Now we need to deliver them to our websocket clients.

Websocket Gateway

Let's take a look at Websocket Gateway

export enum WebsocketEventSubscribeList {
  FETCH_EVENTS_MESSAGES = 'fetch-events-messages',
  EVENTS_MESSAGES_STREAM = 'events-messages-stream',
}

@WebSocketGateway({
  pingInterval: 30000,
  pingTimeout: 5000,
  cors: {
    origin: '*',
  },
})
export class MessagesWebsocketGateway {
  constructor(
    @Inject(EVENT_SUBSCRIBER_TOKEN)
    private eventSubscriber: EventSubscriberInterface,
  ) {}

  @SubscribeMessage(WebsocketEventSubscribeList.FETCH_EVENTS_MESSAGES)
  async streamMessagesData(@ConnectedSocket() client: any) {
    const stream$ = this.createWebsocketStreamFromEventFactory(
      client,
      this.eventSubscriber,
      NewMessageEvent.publishableEventName,
    );

    const event = WebsocketEventSubscribeList.EVENTS_MESSAGES_STREAM;
    return from(stream$).pipe(map((data) => ({ event, data })));
  }

  private createWebsocketStreamFromEventFactory(
    client: any,
    eventSubscriber: EventSubscriberInterface,
    eventName: string,
  ): Observable<any> {
    return new Observable((observer) => {
      const dynamicListener = (message: PublishableEventInterface) => {
        observer.next(message);
      };

      eventSubscriber.on(eventName, dynamicListener);

      client.on('disconnect', () => {
        eventSubscriber.off(eventName, dynamicListener);
      });
    });
  }
}

So there is a thing, in the constructor, we have EVENT_SUBSCRIBER_TOKEN, which type is EventSubscriberInterface. But what it really does? This is how it looks under the hood


export class EventEmitter2EventSubscriber implements EventSubscriberInterface {
  constructor(private eventEmitter: EventEmitter2) {}

  on(name: string, listener: any): void {
    this.eventEmitter.on(name, listener);
  }

  off(name: string, listener: any): void {
    this.eventEmitter.removeListener(name, listener);
  }
}

It's just a wrapper for EventEmitter2, that we are using in the method createWebsocketStreamFromEventFactory

 private createWebsocketStreamFromEventFactory(
    client: any,
    eventSubscriber: EventSubscriberInterface,
    eventName: string,
  ): Observable<any> {
    return new Observable((observer) => {
      const dynamicListener = (message: PublishableEventInterface) => {
        observer.next(message);
      };

      eventSubscriber.on(eventName, dynamicListener);

      client.on('disconnect', () => {
        eventSubscriber.off(eventName, dynamicListener);
      });
    });
  }
}

We are using this wrapped EventEmitter2 to create dynamic listeners on publishableName when websocket clients connect and remove on disconnect.

Then, we are doing nothing more than creating rxjs stream to keep the websocket connection and send messages from the listener via observer.next(message); when a new message occurs.

How this event will reach our listeners?

If you return to the first snippet of code, our Redis pub sub module, then you can spot this in the registerEvents method


 for (const eventPublishableName of eventsPublishableNames) {
              await client.subscribe(eventPublishableName, (message) => {
                const normalizedMessage = JSON.parse(
                  message,
                ) as PublishableEventInterface;
                delete (
                  normalizedMessage as Writeable<PublishableEventInterface>
                ).publishableEventName;
                eventEmitter.emit(eventPublishableName, normalizedMessage);
              });

Which basically listens for events on the pub queue, and then re-emit them via the event emitter.

So, let's sum up what we have done here

We are still using our events in the system like we used to via EventEmitter2, but if we want to publish to our connected websocket clients, then all we have to do is implement PublishableInterface
We are not creating new Redis connections on each connected websocket client
We can scale up our system to X instances and it will still behave in the same way - each connected client will get a copy of the event via websocket, no matter to which instance they will be connected

Working code and example are available here: https://github.com/axotion/nestjs-events-websocket

How to make JWT more secure in NestJS

John Doe — Sat, 21 Oct 2023 15:42:55 +0000

Recently, I've been thinking about how to make some of my crucial endpoints more secure when using locally stored JWT.

It may not be the best practice for security (because of the possibility of XSS attacks), but this was a requirement not set by me. I had to adapt.

So in order to make this more secure, I've found a solution, that, hopefully, will help you as well.

The Problem

I assume that we all know what JWT is - a token issued by the backend service, which cannot be modified by the frontend, because that would change the signature and automatically invalidate the token.

It sounds great, till we meet the situation where the token may be stolen - like in my case with XSS attack. Many websites store tokens in local storage instead of HTTP-only cookie, so they are vulnerable as well.

The Solution

So, when it's not possible to change the way frontend stores that token, we have to change the way we issue and validate the token, and this is the moment when I want to introduce you to a simple solution - JWT with hashed fingerprint, but let's see the code

Let's assume, that we have an endpoint to issue tokens, let's call it sign-in endpoint


@Controller('v1/sign-in')
export class SignInAction {
  constructor(private jwtService: JwtService) {}

  @Post()
  async handle(
    @Req() request: Request,
    @Body() body: SignInHttpRequest,
  ): Promise<SignInHttpResponse> {
    const ip =
      request.headers['x-forwarded-for'] || request.socket.remoteAddress;

    const userAgent = request.headers['user-agent'];

    const token = this.jwtService.sign({
      email: body.email,
      fingerprint: getSHA512Hash(`${ip}${userAgent}`), // It's worth to mention, that to make it even more secure, you should add salt
    });

    return {
      token,
    };
  }
}

So what we are doing here, is basically collecting user agent and remote IP from user that is connecting, hashing it, and then putting it into JWT token, so even if user or attacker would like to see what's inside JWT token, they will only see some random hashed value

But how it's gonna boost my security you ask? Let's take a look at our second endpoint, show the fingerprint

@Controller('v1/fingerprint')
@UseGuards(AuthGuard(JWT_STRATEGY))
export class ShowFingerPrintAction {
  @Get()
  async handle(@Req() request) {
    return {
      fingerprint: request.user.fingerprint,
    };
  }
}

At first, it's nothing special. Endpoint that returns fingerprint from request, so what makes it more secure? The JWT_STRATEGY!

export const JWT_STRATEGY = 'JWT';

@Injectable()
export class JwtStrategy extends PassportStrategy(Strategy, JWT_STRATEGY) {
  constructor() {
    super({
      jwtFromRequest: ExtractJwt.fromAuthHeaderAsBearerToken(),
      ignoreExpiration: false,
      secretOrKey: 'hard!to-guess_secret',
      passReqToCallback: true,
    });
  }

  async validate(request: Request, payload: any): Promise<any> {
    const fingerprint = payload.fingerprint;

    const ip =
      request.headers['x-forwarded-for'] || request.socket.remoteAddress;

    const userAgent = request.headers['user-agent'];
    const calculatedFingerprint = getSHA512Hash(`${ip}${userAgent}`); // It's worth to mention, that to make it even more secure, you should add salt

    if (fingerprint !== calculatedFingerprint) {
      throw new BadRequestException('Invalid fingerprint');
    }

    return payload;
  }
}

As you can see, in strategy, we are extracting the same values as when we extracted them from the sign-in endpoint and comparing them - if the IP or user agent has changed, then we will deny access to our fingerprint endpoint.

This is the case where even if the attacker stole our token, he can't do anything with it, because

He doesn't know what the hash inside the fingerprint is
We won't allow him to connect to our endpoints, because of different IP and user agent (and maybe more factors)

That's how we can protect our crucial endpoint in a very simple way, but of course, there are some tradeoffs

For every call, we have to calculate the sha512 hash
If the user has a dynamic IP he will have to log in frequently

For the first problem, you could of course check only for IP and don't even hash it when issuing a token, but then the potential attacker knows the vector of the attack (IP spoofing or other methods), so it's safer to sacrifice some performance for security.

I hope that this method will help you to introduce an additional layer of security to your crucial endpoints.

Link to working source code: Source Code

UPDATE:

I've added comments about additional salt for sha512 calculations, it makes it less reversible
X-FORWARDER-FOR header can be easily spoofed unless you use services like Cloudflare, keep that in mind

What's the main problem with evaluating your skills?

John Doe — Sat, 16 Jan 2021 22:53:54 +0000

I often doubt my programming skills and experience when it comes to project something unusual or choosing between data transport for a particular case (pub/sub or maybe a queue?).

Last time I caught myself on something trivial (It was 2 years since I had to write it myself) - quicksort. I didn't quite remember how to do it well, so I google formula and write a solution, but again - I started to doubt my skills because of that.

Why I forget about something that simple? Maybe I should write the code and eventually find out the right formula by myself? It would take some time, yeah, but... it would calm my mind, but then I probably would be less productive that day.

I started to blame myself for that later, but after some time I realized that I saw lately something similar - someone on Facebook group bullied junior because he didn't remember the formula for calculating the average from numbers.

I started to imagine - what if the junior and the guy who bullied him was one and the same person?

And then it struck me - Why I should blame myself or feel less confident about my skills when I forget something that I didn't use in a long time?

It will be okay if I don't remember the minor differences between queue or pubsub, because I don't need to remember it! I only need to remember that some minor differences exist - that's all, because if I need to know these differences - then I will just do my research!

I hope it will help someone with similar thoughts

Why I always use CQRS pattern when I work with juniors? 🤔

John Doe — Fri, 13 Mar 2020 18:28:43 +0000

Hi!

This is my second post on dev site. I hope you will enjoy my article about pure CQRS pattern. 😊

A year ago I heard about buzzword called "CQRS". I started my research and after some weeks I catch myself on using it everywhere. I thought it was good idea, but after some time I seen cons and my bad decisions.

So, the time has passed and now I'm ready to talk about it with some advices for newbies in CQRS.

At first, many peoples talks about not using it everywhere and I partially agree with them, but the point is that you can always use only CQRS without DDD, layers and others.
They are ofter goes in pair and I think this is the main purpose of problem.

Why CQRS is good for project with juniors?

As I mid, I have some experience with juniors. They often try to make something work, but not maintainable in long time.
For example, if you give junior task "create CRUD" then it probably would be a mess.

A god controller for everything with many private functions (Of course you should inform juniors why it's bad and teach them to do it in proper way!), but if you introduce CQRS and say "Hey, can you create use case with command and handler in the way I did before?" it will be more easier for junior and you.
It will not ruin the entire system and you can test it without magic, cuz logic will remain in handler.

So, then I should explain idea of CQRS to juniors?

Of course! These juniors deserve it. They are not some zombie coders, and it's good opportunity to teach them about good architecture.

Ok ok, so how do you usually do it?

I try to start with minimal environment. Just actions (single method from controller per file), commands, queries and handlers. I try not to make models in "domain" too smart if it's not required. They can remain anemic for now. And that's all.

You are not supposed to do more about it, but of course I recommend to use full of DDD/Hexanogal/buzzwords if you can see profits from it.

Thanks for read!

In my opinion good sources for CQRS, DDD and hexagonal architecture.

https://martinfowler.com/bliki/CQRS.html

https://herbertograca.com/2017/11/16/explicit-architecture-01-ddd-hexagonal-onion-clean-cqrs-how-i-put-it-all-together/

Why you shouldn`t use axios interceptors with Redux

John Doe — Sat, 29 Feb 2020 15:18:10 +0000

Hi folks. This is my first post here! I hope you will enjoy my thoughts of software best practices 😊

What are the axios interceptors?

They are like simple middlewares which executes some logic before request and after responses. Interceptors are mostly used in authetication logic when we can attach some headers to every request.

axiosInstance.interceptors.request.use(
  config => {
    config.headers.authorization = store.get("authToken");
    return config;
  },
  error => Promise.reject(error)
);

In react projects I often see that programmers couple redux actions with axios interceptors, so for example action loginUser can update state of store and assign authorization token to request interceptor, and on logoutUser can remove token from store and also from axios interceptors. Sounds fine? Then I will explain why not.

At first, they coupled axios and redux. Ok, but… why redux should know details about http client? And what if you want to change http client that not support interceptors? OR what if you want to use the same axios to request to another source? Do you really want to expose your auth headers? 🤔

Ok ok, you MAYBE have right… but what’s a good way to do it?

My propose to deal with it, is a magic thing called HoC. HoC is a good way to decouple your main logic from implementation details.

Your HoC can have redux state with some auth token. HoCcan dynamically set header for every request in your API Client/DataProvider without using interceptors, and then just pass your DataProvider to wrapped component, so your component can be now well tested with mocked DataProvider and without knowing anything about interceptors implementation.

The best source of HoC and example of data provider is in the official react docs https://en.reactjs.org/docs/higher-order-components.html but you can create something similar in almost every framework