Forem: tkssharma

Microservices Design Aspects

tkssharma — Sun, 22 Sep 2024 08:57:28 +0000

Microservices Design Aspects: A Comprehensive Guide

Microservices architecture has gained significant popularity in recent years due to its ability to build scalable, maintainable, and resilient applications. By breaking down large monolithic applications into smaller, independent services, microservices architecture offers several advantages.

Key Design Aspects of Microservices

Bounded Contexts: Define clear boundaries between microservices based on business domains or functionalities. This helps isolate changes and improve maintainability.
API-First Approach: Design microservices with well-defined APIs that clearly communicate their capabilities and contracts.
Decentralized Governance: Allow each microservice team to make independent decisions about technology stacks and development practices.
Asynchronous Communication: Utilize asynchronous communication patterns like messaging queues to decouple microservices and improve scalability.
Data Management: Consider how to manage data across multiple microservices, such as using event sourcing or CQRS.
Deployment and Scaling: Implement strategies for deploying and scaling microservices independently.
Monitoring and Observability: Use monitoring tools to track performance, identify issues, and ensure the overall health of your microservices.

Benefits of Microservices Architecture

Scalability: Each microservice can be scaled independently based on demand.
Resilience: Failures in one microservice don't necessarily affect the entire application.
Technology Agnostic: Microservices can be built using different technologies.
Continuous Delivery: Deploy and update microservices independently, enabling faster development cycles.
Team Autonomy: Empower teams to own and develop their own microservices.

Challenges of Microservices Architecture

Increased Complexity: Managing multiple microservices can be more complex than a monolithic architecture.
Distributed Transactions: Coordinating transactions across multiple microservices can be challenging.
Network Latency: Communication between microservices can introduce latency.
Testing and Debugging: Testing and debugging microservices can be more complex due to their distributed nature.

Best Practices for Microservices Design

Start Small: Begin with a small number of microservices and gradually add more as your application grows.
Use Domain-Driven Design: Apply domain-driven design principles to model your business domains and create well-defined microservices.
Choose the Right Technology Stack: Select the appropriate technology stack for each microservice based on its requirements.
Implement Strong Testing: Ensure code quality and reliability through thorough testing.
Monitor and Optimize: Use monitoring tools to track performance and identify bottlenecks.

By carefully considering these design aspects and best practices, you can successfully implement a microservices architecture that meets the needs of your application and provides numerous benefits.

Building Scalable Microservices with NestJS and Kafka

tkssharma — Sun, 22 Sep 2024 08:57:14 +0000

Building Scalable Microservices with NestJS and Kafka

Introduction

NestJS, a progressive Node.js framework, and Kafka, a distributed streaming platform, form a powerful combination for building scalable and real-time microservices. This guide will explore how to leverage these technologies to create efficient and resilient applications.

Understanding Microservices

Microservices architecture involves breaking down large applications into smaller, independent services that communicate via APIs. This approach offers several benefits, including:

Scalability: Each microservice can be scaled independently based on demand.
Resilience: Failures in one microservice don't necessarily affect the entire application.
Technology Agnostic: Microservices can be built using different technologies.

Introducing Kafka

Kafka is a distributed streaming platform that is ideal for handling high-volume, real-time data streams. It offers features like:

Pub/Sub Model: Producers publish messages to topics, and consumers subscribe to those topics.
Durability: Messages are persisted to disk, ensuring they are not lost.
Scalability: Kafka clusters can be easily scaled horizontally to handle increasing workloads.

Integrating NestJS with Kafka

NestJS provides a convenient way to integrate with Kafka using the @nestjs/microservices package. This package allows you to create microservices that communicate using various protocols, including Kafka.

Creating a Kafka Producer

To create a Kafka producer in NestJS, you can use the following code:

import { KafkaOptions, Transport } from '@nestjs/microservices';

@Injectable()
export class KafkaProducerService {
  private readonly kafkaClient: ClientKafka;

  constructor() {
    this.kafkaClient = new ClientKafka({
      transport: Transport.KAFKA,
      options: {
        client: {
          brokers: ['localhost:9092'],
        },
      },
    });
  }

  async send(message: any) {
    await this.kafkaClient.emit('topic-name', message);
  }
}

Creating a Kafka Consumer

To create a Kafka consumer in NestJS, you can use the following code:

import { KafkaOptions, Transport } from '@nestjs/microservices';

@Injectable()
export class KafkaConsumerService {
  private readonly kafkaClient: ClientKafka;

  constructor() {
    this.kafkaClient = new ClientKafka({
      transport: Transport.KAFKA,
      options: {
        client: {
          brokers: ['localhost:9092'],
        },
      },
    });
  }

  @EventPattern('topic-name')
  async handleEvent(message: any) {
    // Process the message
  }
}

Best Practices for Microservices with Kafka

Design for Scalability: Ensure your microservices are designed to be scalable and can handle increasing workloads.
Use Asynchronous Communication: Utilize Kafka's asynchronous messaging capabilities to decouple microservices.
Handle Failures Gracefully: Implement retry mechanisms and error handling strategies to ensure reliability.
Monitor and Optimize: Use monitoring tools to track performance and identify bottlenecks.

Conclusion

By combining NestJS and Kafka, you can build highly scalable and resilient microservices that can handle real-time data processing and event-driven architectures. Following the best practices outlined in this guide will help you create successful microservices applications.

nestjs with kafkajs Producer and Consumer

tkssharma — Sun, 22 Sep 2024 08:53:04 +0000

NestJS with Kafkajs: A Powerful Combination for Building Scalable Applications

https://github.com/tkssharma/nestjs-kafka-monorepo

Introduction

Kafka is a distributed streaming platform that is widely used for real-time data processing and messaging. NestJS, on the other hand, is a progressive Node.js framework for building scalable and efficient applications. Combining Kafka with NestJS can create a powerful and scalable solution for various use cases.

Integrating Kafka with NestJS

To integrate Kafka with NestJS, we can use the @nestjs/microservices package. This package provides a convenient way to create microservices that communicate using different protocols, including Kafka.

Creating a Kafka Producer

To create a Kafka producer in NestJS, we can use the @nestjs/microservices module and the KafkaOptions interface. Here's an example:

import { KafkaOptions, Transport } from '@nestjs/microservices';

@Injectable()
export class KafkaProducerService {
  private readonly kafkaClient: ClientKafka;

  constructor() {
    this.kafkaClient = new ClientKafka({
      transport: Transport.KAFKA,
      options: {
        client: {
          brokers: ['localhost:9092'],
        },
      },
    });
  }

  async send(message: any) {
    await this.kafkaClient.emit('topic-name', message);
  }
}

Creating a Kafka Consumer

To create a Kafka consumer in NestJS, we can use the @nestjs/microservices module and the KafkaOptions interface. Here's an example:

import { KafkaOptions, Transport } from '@nestjs/microservices';

@Injectable()
export class KafkaConsumerService {
  private readonly kafkaClient: ClientKafka;

  constructor() {
    this.kafkaClient = new ClientKafka({
      transport: Transport.KAFKA,
      options: {
        client: {
          brokers: ['localhost:9092'],
        },
      },
    });
  }

  @EventPattern('topic-name')
  async handleEvent(message: any) {
    // Process the message
  }
}

Using Kafka Producer and Consumer

Once you have created the producer and consumer, you can use them to send and receive messages. Here's an example:

@Controller('kafka')
export class KafkaController {
  constructor(private readonly kafkaProducerService: KafkaProducerService) {}

  @Post()
  async sendMessage(@Body() message: any) {
    await this.kafkaProducerService.send(message);
    return { message: 'Message sent successfully' };
  }
}

Integrate nestjs with simple kafkajs library

import { Module } from '@nestjs/common';
import { KafkaService } from './kafka.service';
import { AppConfigModule } from '@app/config';
import { KafkaConsumerService } from './kafka.consumer.service';
import { KafkaProducerService } from './kafka.producer.service';

@Module({
  imports: [AppConfigModule],
  providers: [KafkaConsumerService, KafkaProducerService],
  exports: [KafkaConsumerService, KafkaProducerService],
})
export class KafkaModule { }

import { AppConfigService } from '@app/config';
import { Injectable, OnApplicationShutdown } from '@nestjs/common';
import { ConfigService } from '@nestjs/config';
import { ConsumerConfig, ConsumerSubscribeTopic, KafkaMessage } from 'kafkajs';
import { KafkaConsumer } from './consumer.service';

interface KafkajsConsumerOptions {
  topic: ConsumerSubscribeTopic;
  config: ConsumerConfig;
  onMessage: (message: KafkaMessage) => Promise<void>;
}

export interface IConsumer {
  connect: () => Promise<void>;
  disconnect: () => Promise<void>;
  consume: (message: any) => Promise<void>;
}


@Injectable()
export class KafkaConsumerService implements OnApplicationShutdown {
  private readonly consumers: IConsumer[] = [];

  constructor(
    private readonly configService: AppConfigService,
  ) { }

  async consume({ topic, config, onMessage }: KafkajsConsumerOptions) {
    const consumer = new KafkaConsumer(
      topic,
      config,
      this.configService.kafka.broker
    );
    await consumer.connect();
    await consumer.consume(onMessage);
    this.consumers.push(consumer);
  }

  async onApplicationShutdown() {
    for (const consumer of this.consumers) {
      await consumer.disconnect();
    }
  }
}

Consumer service

import { Consumer, ConsumerConfig, ConsumerSubscribeTopic, Kafka, KafkaMessage, Producer } from "kafkajs";
import { IProducer } from "./kafka.producer.service";
import { Logger } from "@nestjs/common";
import { IConsumer } from "./kafka.consumer.service";
import * as retry from 'async-retry';

export const sleep = (timeout: number) => {
  return new Promise<void>((resolve) => setTimeout(resolve, timeout));
};

export class KafkaConsumer implements IConsumer {
  private readonly kafka: Kafka;
  private readonly consumer: Consumer;
  private readonly logger: Logger;

  constructor(
    private readonly topic: ConsumerSubscribeTopic,
    config: ConsumerConfig,
    broker: string) {
    this.kafka = new Kafka({
      brokers: [broker]
    })
    this.consumer = this.kafka.consumer(config);
    this.logger = new Logger(`${topic}-${config.groupId}`)
  }
  async consume(onMessage: (message: KafkaMessage) => Promise<void>) {
    await this.consumer.subscribe(this.topic)
    await this.consumer.run({
      eachMessage: async ({ message, partition }) => {
        this.logger.debug(`Processing message partition: ${partition}`);
        try {
          await retry(async () => onMessage(message), {
            retries: 3,
            onRetry: (error, attempt) =>
              this.logger.error(
                `Error consuming message, executing retry ${attempt}/3...`,
                error,
              ),
          });
        } catch (err) {
          // handle failure of message 
          // write then to DB table or log them 
          // better write to DATABASE 
        }
      },
    })
  }

  async connect() {
    try {
      await this.consumer.connect();
    } catch (err) {
      this.logger.error('Failed to connect to Kafka. trying again ...', err);
      await sleep(5000);
      await this.connect();
    }

  }
  async disconnect() {
    this.consumer.disconnect()
  }
}

Conclusion

By integrating Kafka with NestJS, you can build scalable and distributed applications that can handle high volumes of data and real-time processing. The @nestjs/microservices module provides a convenient way to interact with Kafka, making it easy to incorporate into your NestJS projects.

Nestjs Workspaces to build Monorepo

tkssharma — Sun, 22 Sep 2024 08:52:54 +0000

NestJS Workspaces: A Powerful Tool for Monorepo Development

Introduction

In modern software development, managing multiple projects within a single repository, known as a monorepo, has become increasingly popular. It offers several advantages, including improved code sharing, simplified dependency management, and faster build times. NestJS Workspaces is a powerful tool that simplifies the process of building monorepo-based NestJS applications.

What is NestJS Workspaces?

NestJS Workspaces is a feature within the NestJS framework that allows you to manage multiple NestJS applications within a single repository. It provides a convenient way to share code, dependencies, and configuration between different projects, making it easier to maintain and develop complex applications.

Key Benefits of Using NestJS Workspaces:

Improved Code Sharing: Easily share code and libraries between different applications within the workspace.
Simplified Dependency Management: Manage dependencies at the workspace level, ensuring consistency across all projects.
Consistent Development Environment: Maintain a consistent development environment for all applications in the workspace.
Faster Builds and Testing: Leverage caching and parallel execution to improve build and test times.
Enhanced Collaboration: Facilitate collaboration among team members by working on multiple projects within a single repository.

Setting Up a NestJS Workspace

To set up a NestJS workspace, follow these steps:

Create a New Workspace: Use the nest new workspace-name command to create a new workspace.
Generate Applications: Use the nest generate app app-name command to generate individual applications within the workspace.
Share Code: Create shared libraries or modules within the workspace and import them into your applications.

Managing Dependencies

NestJS Workspaces uses a shared package.json file to manage dependencies for all applications within the workspace. This simplifies dependency management and ensures consistency across projects.

Building and Testing

You can use the nest run build and nest test commands to build and test all applications within the workspace. NestJS Workspaces automatically detects changes and rebuilds only the affected applications, improving build times.

Deployment

You can deploy your NestJS applications from the workspace using your preferred deployment strategy. Consider using tools like Docker, Kubernetes, or serverless platforms for efficient deployment.

Conclusion

NestJS Workspaces is a valuable tool for building and managing complex NestJS applications. By leveraging the benefits of monorepo, you can improve code sharing, simplify dependency management, and streamline your development workflow. Consider using NestJS Workspaces in your next NestJS project to experience its advantages firsthand.

DiagramGPT Review of Eraser AI

tkssharma — Sun, 22 Sep 2024 08:52:42 +0000

Eraser AI: A Powerful Tool for Creating Stunning Diagrams

Eraser AI is a revolutionary AI-powered tool that simplifies the process of creating professional-looking diagrams. With its advanced capabilities, Eraser AI can quickly generate various types of diagrams, from flowcharts and UML diagrams to mind maps and network diagrams.

Key Features of Eraser AI:

Natural Language Processing: Eraser AI understands natural language prompts, allowing you to describe your diagram concept in plain English.
AI-Powered Generation: The tool leverages AI algorithms to generate accurate and visually appealing diagrams based on your input.
Customization Options: Easily customize your diagrams with different shapes, colors, and styles to match your preferences.
Export and Sharing: Export your diagrams in various formats (e.g., PNG, SVG, PDF) and share them with others.
Integration with Other Tools: Eraser AI can seamlessly integrate with your existing design workflow.

How to Use Eraser AI:

Access the Tool: Visit the Eraser AI website or download the desktop app.
Describe Your Diagram: Use natural language to describe the type of diagram you want to create.
Customize the Diagram: Modify the generated diagram to suit your specific needs.
Export and Share: Export your diagram in your desired format and share it with others.

Benefits of Using Eraser AI:

Save Time and Effort: Create professional-looking diagrams in a fraction of the time it would take to draw them manually.
Improve Accuracy: Ensure that your diagrams are technically correct and visually appealing.
Enhance Collaboration: Share and collaborate on diagrams with your team members.
Streamline Your Workflow: Integrate Eraser AI into your existing design process.

Use Cases for Eraser AI:

Software Architecture: Create UML diagrams to visualize your software architecture.
Data Flow Diagrams: Illustrate data flow within your system.
Business Process Diagrams: Model your business processes and workflows.
Mind Mapping: Create visual representations of ideas and concepts.
Technical Documentation: Generate diagrams to accompany technical documentation.

Eraser AI is a powerful tool that can significantly improve your efficiency and productivity when creating diagrams. By leveraging AI, Eraser AI automates the process, allowing you to focus on your ideas and concepts. Give Eraser AI a try and experience the benefits of AI-powered diagram creation for yourself!

Running Kafka Locally like a Pro

tkssharma — Tue, 10 Sep 2024 15:33:54 +0000

Running Kafka Locally: A Step-by-Step Guide

I am writing this because on the web there is no proper way to bootstrap kakfa locally and i spent few hours to figure out an issue !! frustrating right !

Introduction

Kafka is a powerful distributed streaming platform that is widely used for real-time data processing and messaging. To get started with Kafka without the need for a cloud-based environment, you can run it locally on your development machine. This guide will walk you through the process of setting up a local Kafka cluster using Docker Compose.

Prerequisites

Docker installed on your system

Steps:

Create a Docker Compose File: Create a file named docker-compose.yml with the following content:

version: '3'
services:
  zookeeper:
    image: wurstmeister/zookeeper:latest
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka:2.11-1.1.1
    ports:
      - "9092:9092"
    links:
      - zookeeper
    environment:
      KAFKA_ADVERTISED_HOST_NAME: ${HOST_IP}
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
      KAFKA_DELETE_TOPIC_ENABLE: 'true'
      KAFKA_CREATE_TOPICS: "topic-test:1:1"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

KAFKA_ADVERTISED_HOST_NAME: ${HOST_IP}

This is very Important line while setting up kafka localy, this will allow local system to use IP and connect to kafka container, so get your IP quickly ! we can HOST_IP while booting up docker compose

Start the Kafka Cluster: Run the following command in your terminal:

  export HOST_IP=$(ifconfig | grep -E "([0-9]{1,3}\.){3}[0-9]{1,3}" | grep -v 127.0.0.1 | awk '{ print $2 }' | cut -f2 -d: | head -n1)
docker-compose up

This will start the Zookeeper and Kafka containers in the background.

Using Kafka

Once Kafka is running, you can produce and consume messages using a Kafka client library. Here's an example using the kafkajs library:

const { Kafka } = require('kafkajs')
const ip = require('ip')

const host = process.env.HOST_IP || ip.address()
console.log(host)

const kafka = new Kafka({
  clientId: 'my-app',
  brokers: [`${host}:9092`],
})

const producer = kafka.producer()
const consumer = kafka.consumer({ groupId: 'test-group' })

const run = async () => {
  // Producing
  await producer.connect()
  await producer.send({
    topic: 'topic-test',
    messages: [
      { value: 'Hello KafkaJS user!' },
    ],
  })

  // Consuming
  await consumer.connect()
  await consumer.subscribe({ topic: 'test-topic', fromBeginning: true })

  await consumer.run({
    eachMessage: async ({ topic, partition, message }) => {
      console.log({
        partition,
        offset: message.offset,
        value: message.value.toString(),
      })
    },
  })
}

run().catch(console.error)

Additional Tips:

Kafka Connect: Use Kafka Connect to integrate with other data sources and sinks.
Schema Registry: Implement a schema registry to ensure data consistency and compatibility.
Security: Configure security measures like authentication and authorization for your Kafka cluster.

By following these steps, you can set up a local Kafka cluster and start experimenting with its capabilities.

Dockerize nestjs application with Postgres

tkssharma — Tue, 10 Sep 2024 07:05:28 +0000

Dockerize NestJS Application with Postgres
This video is Part of this series "Scale Nodejs Nestjs with Prometheus & Grafana Metrics" In this part of video we are creating container for nestjs microservice and spinning postgres container with it

We are covering

Dockerfile for nestjs application
Creating nestjs container
Spinning postgres container
Running Migration in nestjs container

Part-1
https://www.youtube.com/watch?v=cU0kdflSMzg
Part-2
https://www.youtube.com/watch?v=yPJGDaGUIIE
Part-3
https://www.youtube.com/watch?v=Kub95iN68_w
Part-4
https://www.youtube.com/watch?v=2NCiNC5z2Yc

Dockerizing Your NestJS Application with PostgreSQL
Simplify Deployment and Management with Docker

Docker has revolutionized the way we deploy and manage applications. In this tutorial, you'll learn how to containerize your NestJS application and integrate it with PostgreSQL using Docker Compose.

Key Topics Covered:

Understanding Docker: Learn the basics of Docker containers and images.
Creating a Dockerfile: Define your NestJS application's environment using a Dockerfile.
Setting Up PostgreSQL: Create a Docker container for your PostgreSQL database.
Using Docker Compose: Manage both your NestJS application and PostgreSQL database using Docker Compose.
Connecting to PostgreSQL: Configure your NestJS application to connect to the PostgreSQL database.
running typeorm migration on container
Deployment: Deploy your Dockerized application to a production environment.

Benefits of Dockerizing Your NestJS Application:

Portability: Run your application consistently across different environments.
Isolation: Ensure your application is isolated from other running services.
Scalability: Easily scale your application horizontally by adding more containers.
Reproducibility: Create consistent development and production environments.

Join us on this journey to master Docker and streamline your NestJS application deployment.

Keywords: Docker, NestJS, PostgreSQL, containerization, deployment, web development, software development, tutorial

Discover the magic of custom decorators in NestJS #nestjs

tkssharma — Sat, 07 Sep 2024 10:24:17 +0000

Introduction

NestJS, a progressive Node.js framework, offers a rich set of features that make it a popular choice for building scalable and efficient web applications. One of its standout features is the ability to create custom decorators, which provide a powerful and flexible way to extend the framework's functionality and tailor it to your specific needs.

Understanding Decorators

In NestJS, decorators are functions that can be applied to classes, methods, or properties to modify their behavior. They provide a declarative approach to adding functionality to your code, making it more concise and readable.

Creating Custom Decorators

To create a custom decorator, you can use the following steps:

Define a decorator function: Create a function that takes a target, propertyKey, and descriptor as arguments.
Modify the target, propertyKey, or descriptor: Implement the desired behavior within the decorator function.
Apply the decorator: Use the @ syntax to apply the decorator to the target element.

Example:

import { Injectable, UseGuards } from '@nestjs/common';

function MyCustomDecorator(message: string) {
  return (target: any, propertyKey: string, descriptor: PropertyDescriptor) => {
    console.log(`Applying decorator to ${target.constructor.name}.${propertyKey}`);
    // Modify the descriptor here
  };
}

@Injectable()
export class MyService {
  @MyCustomDecorator('Hello')
  myMethod() {
    // ...
  }
}

Common Use Cases for Custom Decorators

Validation: Create custom validators to enforce specific rules on input data.
Authorization: Implement role-based or permission-based authorization.
Logging: Log information about function calls and their arguments.
Caching: Cache results of expensive operations.
Performance monitoring: Measure the performance of specific methods.

Advanced Decorators

NestJS also provides advanced decorator features, such as:

Class decorators: Modify the behavior of entire classes.
Method decorators: Modify the behavior of individual methods.
Property decorators: Modify the behavior of class properties.
Parameter decorators: Modify the behavior of function parameters.

Best Practices

Keep decorators concise and focused.
Use meaningful names for decorators.
Test your custom decorators thoroughly.
Consider using existing decorators or libraries if available.

Conclusion

Custom decorators are a powerful tool in NestJS that allow you to extend the framework's functionality and tailor it to your specific requirements. By understanding how to create and use custom decorators, you can write more efficient, maintainable, and expressive code.

Fix long import paths in your NestJS project #nestjs

tkssharma — Sat, 07 Sep 2024 10:24:14 +0000

Taming the Import Jungle: Simplifying Long Import Paths in NestJS

Introduction

As your NestJS projects grow in size and complexity, managing long import paths can become cumbersome and error-prone. Fortunately, NestJS provides a convenient solution to streamline your imports and enhance code readability: path aliases. In this blog post, we'll explore how to effectively use path aliases to simplify your NestJS project's import structure.

Understanding Path Aliases

Path aliases allow you to create custom shortcuts for frequently used paths within your project. Instead of specifying lengthy relative paths, you can use a shorter alias, making your imports more concise and easier to maintain.

Setting Up Path Aliases in NestJS

Configure tsconfig.json:
- Open your project's tsconfig.json file and add the baseUrl and paths properties:

   {
     "compilerOptions": {
       "baseUrl": "./src",
       "paths": {
         "@app/": ["src/app"],
         "@modules/": ["src/modules"],
         // Add more aliases as needed
       }
     }
   }

Use Aliases in Imports:
- Replace long import paths with their corresponding aliases. For example:

   // Before:
   import { MyService } from './modules/my-module/my.service';

   // After:
   import { MyService } from '@modules/my-module/my.service';

Best Practices for Path Aliases

Choose meaningful aliases: Use clear and concise aliases that reflect the structure of your project.
Avoid circular references: Ensure that your aliases do not create circular dependencies.
Update aliases as needed: As your project evolves, update your aliases to reflect changes in the directory structure.
Use a linter: Configure your linter to enforce consistent path alias usage.

Benefits of Using Path Aliases

Improved readability: Shorter and more concise imports make your code easier to understand.
Reduced errors: Fewer typos and mistakes when dealing with long paths.
Enhanced maintainability: Easier to refactor and reorganize your project.
Consistent naming conventions: Promotes a consistent naming structure throughout your project.

Conclusion

Github
https://github.com/tkssharma/nestjs-advanced-2023/tree/main/apps/23-nestjs-path-mapping

By effectively using path aliases in your NestJS projects, you can significantly improve code readability, maintainability, and reduce the likelihood of errors. This simple technique can have a substantial impact on the overall quality and efficiency of your development process.

Nestjs file upload like a Pro [in depth]

tkssharma — Sat, 07 Sep 2024 08:00:41 +0000

NestJS File Uploads to AWS S3 with Validation: A Comprehensive Guide

Introduction

File uploads are a common feature in many web applications. NestJS, a progressive Node.js framework, provides a convenient way to handle file uploads and integrate with cloud storage services like AWS S3. In this blog post, we'll explore how to implement file uploads with validation in NestJS and store the uploaded files on AWS S3.
sample example i am showing here --

Prerequisites

A NestJS project set up.
An AWS account with an S3 bucket created.
AWS credentials configured in your project's environment variables.

Steps:

Install Required Packages:

   npm install @nestjs/platform-express @nestjs/aws-sdk/s3 multer

Create an S3 Service:

   import { Injectable } from '@nestjs/common';
   import { S3 } from '@nestjs/aws-sdk/s3';

   @Injectable()
   export class S3Service {
     constructor(private readonly s3: S3) {}

     async uploadFile(file: Express.Multer.File): Promise<string> {
       const params = {
         Bucket: 'your-bucket-name',
         Key: file.originalname,
         Body: file.buffer,
         ContentType: file.mimetype,
       };

       const result = await this.s3.upload(params).promise();
       return result.Location;
     }
   }

Configure Multer:

   import { Module } from '@nestjs/common';
   import { MulterModule } from '@nestjs/platform-express';

   @Module({
     imports: [
       MulterModule.register({
         storage: multer.diskStorage({
           destination: './uploads',
           filename: (req, file, cb) => {
             cb(null, `${file.originalname}-${Date.now()}`);
           },
         }),
       }),
     ],
   })
   export class UploadModule {}

Create a Controller:

   import { Controller, Post, UseInterceptors, UploadedFile, UploadedFiles } from '@nestjs/common';
   import { FileInterceptor, FilesInterceptor } from '@nestjs/platform-express';
   import { S3Service } from './s3.service';

   @Controller('upload')
   export class UploadController {
     constructor(private readonly s3Service: S3Service) {}

     @Post('single')
     @UseInterceptors(FileInterceptor('file'))
     async uploadSingleFile(@UploadedFile() file: Express.Multer.File) {
       const url = await this.s3Service.uploadFile(file);
       return { url };
     }

     @Post('multiple')
     @UseInterceptors(FilesInterceptor('files'))
     async uploadMultipleFiles(@UploadedFiles() files: Express.Multer.File[]) {
       const urls = await Promise.all(files.map(file => this.s3Service.uploadFile(file)));
       return { urls };
     }
   }

Code example with Validation

// Native.
/* eslint-disable no-useless-escape */

// Package.
import {
  Body,
  Controller,
  Get,
  HttpCode,
  HttpStatus,
  UploadedFile,
  UseInterceptors,
  ParseFilePipeBuilder,
  Param,
  Post,
  Query,
  UseGuards,
  UsePipes,
  ValidationPipe,
  Req,
  BadRequestException,
  UploadedFiles,
} from '@nestjs/common';
import {
  ApiBearerAuth,
  ApiConsumes,
  ApiCreatedResponse,
  ApiForbiddenResponse,
  ApiInternalServerErrorResponse,
  ApiNotFoundResponse,
  ApiOkResponse,
  ApiOperation,
  ApiTags,
  ApiUnprocessableEntityResponse,
} from '@nestjs/swagger';
import { INTERNAL_SERVER_ERROR, NO_ENTITY_FOUND } from '../../../app.constants';
import { RestaurantService } from '../services/restaurant.service';
import { Type } from 'class-transformer';
import {
  CreateRestaurantBodyDto,
  SearchQueryDto,
  getRestaurantByIdDto,
} from '../dto/restaurant.dto';
import { RolesAllowed } from '../../../core/decorator/role.decorator';
import { Roles } from '../../../core/roles';
import { RolesGuard } from '../../../core/guard/role.guard';
import { FileInterceptor, FilesInterceptor } from '@nestjs/platform-express';
import {
  uploadFile,
  uploadFiles,
} from '../../../core/decorator/file.decorator';
import { imageFileFilter } from '../../../core/filters/file.filter';
import { GetCurrentUser, User, UserData, UserMetaData } from '../interface';

const SIZE = 2 * 1024 * 1024;
const VALID_UPLOADS_MIME_TYPES = ['image/jpeg', 'image/png'];

@ApiBearerAuth('authorization')
@Controller('restaurants')
@UseGuards(RolesGuard)
@UsePipes(
  new ValidationPipe({
    whitelist: true,
    transform: true,
  }),
)
@ApiTags('restaurant')
export class RestaurantController {
  constructor(private readonly service: RestaurantService) {}


  // multiple file upload
  @UseGuards(RolesGuard)
  // add all roles which we want to allow
  @RolesAllowed(Roles['admin'])
  @HttpCode(HttpStatus.OK)
  @ApiConsumes('application/json')
  @ApiNotFoundResponse({ description: NO_ENTITY_FOUND })
  @ApiForbiddenResponse({ description: 'UNAUTHORIZED_REQUEST' })
  @ApiUnprocessableEntityResponse({ description: 'BAD_REQUEST' })
  @ApiInternalServerErrorResponse({ description: INTERNAL_SERVER_ERROR })
  @ApiOperation({
    description: 'search restaurants based on lat/lon',
  })
  @ApiOkResponse({
    description: 'return search restaurants successfully',
  })
  @Get('/:id')
  public async getRestaurantById(@Param() param: getRestaurantByIdDto) {
    return await this.service.getRestaurantById(param);
  }

  // multiple file upload
  @UseGuards(RolesGuard)
  // add all roles which we want to allow
  @RolesAllowed(Roles['admin'])
  // one file upload
  @Post('one-file')
  // custom decorator
  // ONE FILE UPLOAD ONLY
  @uploadFile('file')
  @ApiForbiddenResponse({ description: 'UNAUTHORIZED_REQUEST' })
  @ApiUnprocessableEntityResponse({ description: 'BAD_REQUEST' })
  @ApiInternalServerErrorResponse({ description: INTERNAL_SERVER_ERROR })
  @UseInterceptors(FileInterceptor('file'))
  @ApiConsumes('multipart/form-data')
  public async uploadFileOne(
    @UploadedFile(
      new ParseFilePipeBuilder()
        .addFileTypeValidator({ fileType: /(jpg|png)$/ })
        .addMaxSizeValidator({ maxSize: SIZE })
        .build({
          errorHttpStatusCode: HttpStatus.UNPROCESSABLE_ENTITY,
        }),
    )
    file: Express.Multer.File,
  ) {
    try {
      return file;
    } catch (err) {
      throw err;
    }
  }

  // multiple file upload
  @UseGuards(RolesGuard)
  // add all roles which we want to allow
  // upload many files together P
  @RolesAllowed(Roles['admin'])
  @Post('many-files')
  // custom decorator
  @uploadFiles('file')
  @ApiForbiddenResponse({ description: 'UNAUTHORIZED_REQUEST' })
  @ApiUnprocessableEntityResponse({ description: 'BAD_REQUEST' })
  @ApiInternalServerErrorResponse({ description: INTERNAL_SERVER_ERROR })
  @UseInterceptors(FilesInterceptor('file'))
  // all plural here
  // validation with many files
  @ApiConsumes('multipart/form-data')
  public async uploadFiles(
    @UploadedFiles(
      // it will validate each and every files
      new ParseFilePipeBuilder()
        .addFileTypeValidator({ fileType: /(jpg|png)$/ })
        .addMaxSizeValidator({ maxSize: SIZE })
        .build({
          errorHttpStatusCode: HttpStatus.UNPROCESSABLE_ENTITY,
        }),
    )
    files: Array<Express.Multer.File>,
  ) {
    try {
      // once we have files lets upload them in a Loop
      console.log(files);
      return await this.service.upload(files);
    } catch (err) {
      throw err;
    }
  }

  // multiple file upload
  @UseGuards(RolesGuard)
  // add all roles which we want to allow
  @RolesAllowed(Roles['admin'])
  @Post('custom-validation')
  // custom decorator
  @uploadFile('file')
  @ApiForbiddenResponse({ description: 'UNAUTHORIZED_REQUEST' })
  @ApiUnprocessableEntityResponse({ description: 'BAD_REQUEST' })
  @ApiInternalServerErrorResponse({ description: INTERNAL_SERVER_ERROR })
  @UseInterceptors(
    FileInterceptor('file', {
      fileFilter: imageFileFilter,
    }),
  )
  @ApiConsumes('multipart/form-data')
  public async uploadFilev2(@Req() req: any, @UploadedFile() file) {
    console.log(file);
    if (!file || req.fileValidationError) {
      throw new BadRequestException(
        'invalid file provided, allowed *.pdf single file for Invoice',
      );
    }
    return file;
  }
}

Validation:
- Use custom validators or third-party libraries like class-validator to validate uploaded files.
- For example, you can check file size, type, and format.

Additional Considerations:

Error handling: Implement proper error handling to catch exceptions and provide informative messages.
Security: Ensure proper security measures to prevent unauthorized access to uploaded files.
Performance: Consider optimizing file uploads for large files or high traffic.
Scalability: Evaluate your storage solution to ensure it can handle future growth.

By following these steps and incorporating validation, you can effectively handle file uploads in your NestJS applications and store them securely on AWS S3.

Part-1

Part-2

Playlist

Github
https://github.com/tkssharma/nestjs-advanced-2023/tree/main/apps/22-nestjs-file-upload

How to add E2E Tests for Nestjs graphql

tkssharma — Sat, 07 Sep 2024 08:00:10 +0000

Github
https://github.com/tkssharma/nestjs-advanced-2023/tree/main/apps/09.12-nestjs-graphql-e2e-v4

I'd be glad to provide a comprehensive guide on how to add NestJS GraphQL E2E testing, incorporating insights from the ratings and addressing potential shortcomings:

Understanding E2E Testing

End-to-End (E2E) testing simulates real-world user interactions with your application, verifying that it works as expected from start to finish. In the context of NestJS GraphQL, E2E testing involves sending GraphQL queries or mutations to your API and asserting the correctness of the responses.

Setting Up E2E Testing

Install Required Packages:

   npm install @nestjs/testing @nestjs/graphql @nestjs/axios jest @graphql-tools/load-files @graphql-tools/schema

Create an E2E Testing Module:

   import { Module, TestingModuleBuilder } from '@nestjs/testing';
   import { GraphQLModule } from '@nestjs/graphql';
   import { loadSchemaSync } from '@graphql-tools/load-files';
   import { join } from 'path';

   @Module({})
   export class AppModule {}

   export function createTestingModule() {
     return new TestingModuleBuilder()
       .forRootAsync({
         imports: [GraphQLModule.forRoot({
           typeDefs: loadSchemaSync(join(process.cwd(), 'graphql-schemas', '*.graphql')),
           resolvers: { Query: QueryResolver, Mutation: MutationResolver },
         })],
       })
       .compile();
   }

Write E2E Tests:

   import { Test } from '@nestjs/testing';
   import { AppModule } from './app.module';
   import { INestApplication } from '@nestjs/common';
   import { gql } from 'apollo-server-express';
   import * as request from 'supertest';

   describe('E2E', () => {
     let app: INestApplication;

     beforeEach(async () => {
       app = await createTestingModule().createNestApplication();
       await app.init();
     });

     afterEach(async () => {
       await app.close();
     });

     it('should return "Hello world!"', async () => {
       const query = gql`
         query {
           hello
         }
       `;

       const response = await request(app.getHttpServer())
         .post('/graphql')
         .send({ query });

       expect(response.body.data.hello).toBe('Hello world!');
     });
   });

Running E2E Tests

Create a jest.config.js file with the following configuration:

   module.exports = {
     preset: '@nestjs/testing',
     testRunner: 'jest-circus',
     roots: ['<rootDir>/test'],
     testMatch: ['<rootDir>/test/**/*.e2e-spec.ts'],
     transform: {
       '^.+\\.(js|ts|mjs)$': 'ts-jest',
     },
   };

Run the tests using the Jest command:

   npm run test:e2e

Additional Tips

Use a GraphQL client like Apollo Client or urql for more advanced testing scenarios.
Consider using a testing database or mocking external dependencies for isolation.
Write clear and concise test cases to ensure code coverage and maintainability.

By following these steps and incorporating E2E testing into your NestJS GraphQL development process, you can significantly improve the quality and reliability of your application.

why Nestjs is Fav for Backend developers

tkssharma — Fri, 06 Sep 2024 09:33:56 +0000

why Nestjs is Fav for Backend developers

Why NestJS is a Preferred Choice for Backend Developers in 2024

NestJS, a progressive Node.js framework, has gained significant popularity among backend developers in recent years. Its robust features, modular architecture, and strong community support make it an excellent choice for building scalable and maintainable web applications. Let's explore why NestJS continues to be a favored option for backend development in 2024:

1. TypeScript Support:

Improved code quality: TypeScript's static typing helps catch errors early in the development process, leading to more reliable and maintainable code.
Enhanced readability: TypeScript provides better code readability and understanding, especially for large projects.
Strong tooling support: Popular IDEs and code editors offer excellent TypeScript support, including autocompletion, type checking, and refactoring.

2. Modular Architecture:

Code organization: NestJS promotes a modular architecture, allowing you to break down your application into smaller, reusable components. This improves code maintainability and testability.
Dependency injection: NestJS provides a built-in dependency injection system, making it easy to manage dependencies and promote loose coupling between components.

3. Powerful Features:

Routing: NestJS offers a flexible routing system for handling HTTP requests.
Middleware: Use middleware to intercept requests and responses, adding custom logic or handling cross-cutting concerns.
Exception filters: Handle exceptions gracefully and provide informative error messages.
Pipes: Validate and transform data before it reaches your controllers.
Guards: Protect your API endpoints with authentication and authorization mechanisms.

4. Community and Ecosystem:

Active community: NestJS has a large and active community of developers, providing support, resources, and contributions.
Rich ecosystem: A vast ecosystem of third-party modules and libraries is available for NestJS, extending its capabilities.

5. Performance and Scalability:

Optimized for performance: NestJS is designed to be performant, making it suitable for high-traffic applications.
Scalability: The modular architecture and dependency injection system make it easy to scale your NestJS applications as needed.

6. Integration with Angular:

Seamless integration: NestJS integrates seamlessly with Angular, making it a popular choice for full-stack development.

7. Growing Adoption:

Industry recognition: NestJS has gained recognition and adoption from various organizations and companies.
Real-world use cases: NestJS is used in production environments for a wide range of applications.

Conclusion

NestJS continues to be a strong choice for backend developers in 2024 due to its combination of features, performance, and community support. Its modular architecture, TypeScript support, and powerful ecosystem make it an excellent option for building scalable and maintainable web applications.