Forem: admirnisic

Node.js and Kafka as message broker for inter-service communication

admirnisic — Sat, 03 Dec 2022 09:44:09 +0000

In this tutorial we are going to create a simple Node.js application which uses Kafka as a message broker for sending data between instances which act as microservices.

Before we dive into coding let's talk about monolithic (backend) application and microservice architecture. Monolithic application is an application where data access, business logic, UI (sometimes) and other parts of software are combined into a single program and runs on a single platform and is independent from other computing applications.
On the other hand, microservice architecture is an approach for building applications as a set of small independent services which communicate to each other. There are advantages and use cases why microservices should be used instead of monolithic applications. For example, horizontal scalability is a huge advantage where more intensive services can be scaled in order to improve the performance of the service while on monolithic there is only an option to scale the whole application. Services are independend and can be written by different teams and in different programming languages which improves cross team functionality. The failure of one service is less likely to negatively impact other parts of the application because the microservice runs autonomously from the others which is not the case with monolithic applications.

So, considering that the microservice architecture is composed of several independent small services, it is necessary for these services to communicate with each other, and now we are entering into the story with message brokers.

Message brokers and Kafka

By IBM message brokers are an inter-application communication technology to help build a common integration mechanism to support cloud native, microservices-based, serverless and hybrid cloud architectures.

There are a couple of message brokers available and the most popular are Kafka, Redis pub/sub and RabbitMQ.

Let's briefly talk about communication between one-to-one and one-to-many instances. As I already said in the introduction, the good thing about microservice architecture is that horizontal scalability is possible, which means that we can run one service on several machines in order to have faster data processing. Let's imagine for a moment that we have service-a that needs to send data to service-b. If service-b is available on 2 machines and we need service-a to send data to only one service-b instance. Kafka and RabbitMQ are offering instance discovery out-of-the-box, while Redis does not offer service discovery but instead Redis we will send data to both instances of service-b.
Some frameworks such as Moleculer have built-in node (service) discovery, which is in charge of knowing at all times which services are available and which are not, so that the framework itself knows to which instances it is possible to send data.

What needs to be emphasized is that Kafka has the ability to store messages that are sent between instances. If service-a sent a message to service-b but service-b was not available for any reason, the message was already saved and the destination did not recieve the message. When service-b becomes online again, then the service will start receiving messages that were in the queue and that did not arrive at the destination. So, Kafka has the ability to save data and the amount of data to be saved can be set via configuration. Redis, on the other hand, when a service sends a message, only those services that are subscribed at that moment will receive the data.

Kafka

Apache Kafka is a very popular open-source tool and it is a distributed event streaming platform used for high-performance data pipelines, streaming analytics, communication between microservice instances and more. Advantages of using kafka are:

Apache Kafka offers low latency value.
Kafka is able to handle more number of messages of high volume and high velocity.
Kafka has an essential feature to provide resistant to node/machine failure within the cluster.
Apache Kafka contains a distributed architecture which makes it scalable.

Now let's clarify terms that are often used when working with Kafka and those terms are producer, consumer and topic.
The Producer API allows applications to send streams of data to topics.
The Consumer API allows applications to read streams of data from topics.
Topics are partitioned, meaning a topic is spread over a number of 'buckets' located on different Kafka brokers. When a new event is published to a topic, it is actually appended to one of the topic's partitions. Consumers of a specific topic will always read partition's events in exactly the same order they were written. If there is only one partition per topic then there can only be one consumer. If we want to have two consumers per one topic then the topic has to have at least two partitions.

Kafka installation with docker.

Follow the instructions on how to setup a Node.js server with TypeScript in the tutorial which is part of this series. Tutorial 1

After we followed all the instructions from the provided tutorial now we should be able to run a Node.js server with command npm run start which should start a server on port 3000.

Now let's install Kafka. We can install Kafka in several ways, but for me the easiest way is to install it via Docker. We will install the following Docker images:

Kafka
Zookeeper
Kafka UI

In the root folder (link of github repository is provided at the begining of the article), there is a file docker-compose.yml. Let's start the images with command docker-compose up -d and make sure that the containers are up and running.

Installation and server code

Now we have come to the step of dealing with the coding itself. First, we need to install the kafkajs npm package in order to connect the Node.js application with Kafka. Let's install npm package with following command:

npm install --save kafkajs

Now it is necessary to modify the file src/server.ts. Let's create a connection for Kafka.

import express from 'express';
import { Request, Response } from 'express';
import { Kafka, Producer } from 'kafkajs';

const app = express();

// Instance of Kafka
const kafka = new Kafka({
  clientId: 'my-app',
  brokers: ['kafka:9092'],
});

const producer: Producer = kafka.producer();

// Connect to producer
async function runProducer() {
  await producer.connect();
  console.log('Producer connected');
}

runProducer().catch(console.error);

app.get('/', (req: Request, res: Response) => {
  res.send('Application works!');
});

app.listen(3000, () => {
  console.log('Application started on port 3000!');
});

If we look at the code snippet above, we'll see brokers: ['kafka:9092']. With this part of code we will connect to url kafka and port 9092. Since in this case the host is kafka and not localhost, it is necessary to create a proxy that will redirect the network from kafka to localhost. We can achieve this by adding 127.0.0.1 kafka in the file etc/hosts

# file /etc/hosts
127.0.0.1 kafka

Now let's add a producer. As we said in the introduction, the producer is an API that sends messages to a specific topic. We will create a route /send of method GET and whenever we call the mentioned endpoint we will send a message to the topic topic-test-1 which we will create soon.

app.get('/send', async (req: Request, res: Response) => {
  await producer.send({
    topic: 'topic-test-1', // topic name
    messages: [{ value: 'Hello KafkaJS user!' + Math.random().toString() }],
  });

  res.send('Application works!');
});

The idea is to create two consumers that will listen to topic topic-test-1. Let's create a file src/consumer/index.ts with the following code:

import { Kafka } from 'kafkajs';

const kafka = new Kafka({
  clientId: 'my-app',
  brokers: ['kafka:9092'],
});

const consumer = kafka.consumer({ groupId: 'topic-test-1-group' });

consumer.connect();
consumer.subscribe({ topic: 'topic-test-1', fromBeginning: true });

consumer.run({
  // eslint-disable-next-line @typescript-eslint/no-unused-vars
  eachMessage: async ({ topic, partition, message }) => {
    console.log({
      value: message.value.toString(),
    });
  },
});

In package.json add script for running the consumer.

"start:consumer": "npx ts-node ./src/consumer/index.ts"

There is one more step before we start the application. Let's create a topic on Kafka called topic-test-1 with two partitions.
Creating a topic can be done in several ways, and in this tutorial I will show you how to create a topic via the command line tool and via the Kafka UI container that we created via Docker.

With UI
Visit http://localhost:8080 which should open the application for Kafka UI. Go to topics and create a new topic with name topic-test-1 and it should have two partitions.
With command line
Command for creating a topic:

docker exec -it kafka kafka-topics --bootstrap-server localhost:9092 --replication-factor 1 --partitions 2 --create --topic topic-test-1

Command for deleting a topic:

docker exec -it kafka kafka-topics --bootstrap-server localhost:9092 --delete --topic topic-test-1

Now let's start the server so that the producer can send messages to the topic.

npm run start

Now let's start two consumers in two different terminals.

npm run start:consumer

Now it is necessary to send a GET request to the endpoint http://localhost:3000/send. Let's do it with the help of curl with the command curl localhost:3000/send

I have sent three requests and in the image above we can see that we have two consumers running and the first consumer recieved one message and the second consumer recieved two messages. If we add another instance of consumer then one of three instances will be disconnected from Kafka and will not recieve any message while the other two will still be recieving the messages. If we have only one instance then one instance will start recieveing all the messages that are sent to topic topic-test-1.

Wrapping up

In this tutorial we talked about the differences between monolithic application and microservice architecture and what are the advanteges of using microsevice architecture. We talked briefly about what a message broker is, which are the popular message brokers and what are the differences between them. We also talked briefly about Kafka and what are the benefits of using it. And finally, we demonstrated how to use Kafka in a Node.js application. We showed how to connect to Kafka, how to send data via producer API and how to read data on two instances via consumer API.

Real time communication with Socket.IO and Node.js

admirnisic — Sun, 29 May 2022 10:28:10 +0000

Github repository

Socket.IO enables real-time event-based communication in both directions (client and server side). Socket.IO is built on top of WebSockets API and Node.js.

A real-time application (RTA) is an application that functions within a period that the user senses as immediate. Some examples of real-time applications are:

instant messaging system - Chat apps like WhatsApp, Facebook Messenger etc,
push notifications,
applications like Google docs which allows multiple people to update the same document simultaneously.

Writting a real-time application without WebSockets, using HTTP requests, has been traditionally very hard. It involves polling the server for changes and by design it is very slow and requires more resources.

Sockets are an easier and faster solution which most real-time systems are designed on which provides bi-directional communication channel between a client and a server. This means that whenever an event occurs, the server can push messages to clients so the client gets notified immediately and vice versa.

Socket.IO is quite popular and it is used by Microsoft Office, Yammer, Trello...

Installation and server code

Altough this tutorial is provided in series, this specific tutorial is standalone so no previous code needs to be included. Let's start with server code and let's install all the necesarry libraries.

npm install --save socket.io express
npm install --save-dev @types/socket.io

Initial server code:

// ... existing code ...

import http from 'http';
import { Server as SocketIOServer } from 'socket.io';

// Create a new HTTP server for socket.io
const socketServer = http.createServer();
const io = new SocketIOServer(socketServer, {
  cors: {
    origin: "*", // Adjust this to your needs
    methods: ["GET", "POST"]
  }
});
// Listen for socket.io connections
io.on('connection', (socket) => {
  // Listen for socket.io connections
  io.on('connection', (socket) => {
    console.log('New connection created');

    // Get the auth token provided on handshake.
    const token = socket.handshake.auth.token;
    console.log('Auth token', token);

    try {
      // Verify the token here and get user info from JWT token.
    } catch (error) {
      socket.disconnect(true);
    }

    // A client is disconnected.
    socket.on('disconnect', () => {
      console.log('A user disconnected');
    });

    // Read message recieved from client.
    socket.on('message_from_client', (data) => {
      console.log('message_from_client: ', data);
    });

    // Send a message to the connected client 5 seconds after the connection is created.
    setTimeout(() => {
      socket.emit('message_from_server', `Message: ${Math.random()}`);
    }, 5_000);
  });

  // Add more socket event handlers here
});

// Start the socket.io server on a different port, e.g., 4000
socketServer.listen(4000, () => {
  console.log('Socket.io server started on port 4000');
});

// ... existing code ...

app.listen(3000, () => {
  console.log('Application started on port 3000!');
});

In the code snipet above we created an Express server on port 3000 and after that we created a Socket.IO server. socketIo.on('connection', (socket) is called when a new connection from the client side is initiated. This is called a handshake and the first step to do after this is to get the auth token from the client and verify it. If the JWT is malicius then we will disconnect the client and the client will not get any events from the server side and if the token is valid we can get the user data from the JWT.

Sending data from client to server and vice versa is pretty simple.

For reading the data we are using socket.on either from client to server or from server to client.
For sending the data we are using socket.emit either from client to server or from server to client. In the code below we are reading the data from the client side and we are listening to the event message_from_client.

socket.on('message_from_client', (data) => {
  console.log('message_from_client: ', data);
});

Whenever the client is emitting socketIo.emit('message_from_client', 'Sent an event from the client!'); server will read the data in real time.

In the code below we can see how we can send the data from the server to the client with event name message_from_server. Client listening on event message_from_server will read the data in real time.

setTimeout(() => {
  socket.emit('message_from_server', `Message: ${Math.random()}`);
}, 5_000);

Client code

Let's create a simple html file inside src (root) folder with the following code. We will establish client communication with the server by pressing a button.

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Socket tutorial</title>
  </head>
  <body>
    <button onclick="establishConnection()">Join</button>

    <script
      src="https://cdn.socket.io/3.1.3/socket.io.min.js"
      integrity="sha384-cPwlPLvBTa3sKAgddT6krw0cJat7egBga3DJepJyrLl4Q9/5WLra3rrnMcyTyOnh"
      crossorigin="anonymous"
    ></script>
    <script>
      let isConnectionEstablished = false;

      function establishConnection() {
        if (isConnectionEstablished) {
          return;
        }

        isConnectionEstablished = true;

        const socketIo = io('http://localhost:4000', {
          auth: {
            token: 'json-web-token',
          },
        });

        socketIo.on('connect', function () {
          console.log('Made socket connection', socketIo.id);
        });

        socketIo.on('message_from_server', function (data) {
          console.log('message_from_server data: ', data);
        });

        socketIo.on('disconnect', function () {
          console.log('disconnect');
        });

        // Send a message to the server 3 seconds after initial connection.
        setTimeout(function () {
          socketIo.emit('message_from_client', 'Sent an event from the client!');
        }, 3000);

        socketIo.on('connect_error', function (err) {
          console.log('connection errror', err);
        });

      }
    </script>
  </body>
</html>

It is important to note that we provided the Socket.IO client library with script from CDN.

<script
  src="https://cdn.socket.io/3.1.3/socket.io.min.js"
  integrity="sha384-cPwlPLvBTa3sKAgddT6krw0cJat7egBga3DJepJyrLl4Q9/5WLra3rrnMcyTyOnh"
  crossorigin="anonymous"
></script>

Creating a communication channel by sending a token which is required for validating the user.

const socketIo = io('http://localhost:4000', {
  auth: {
    token: 'json-web-token',
  },
});

Code for reading messages from from the server on event message_from_server:

socketIo.on('message_from_server', function (data) {
  console.log('message_from_server data: ', data);
});

Code for sending the data from the client to the server:

setTimeout(function () {
  socketIo.emit('message_from_client', 'Sent an event from the client!');
}, 3000);

Run the application

Now let's run the Node.js application with:

npm run start

Now we can open the index.html file in any browser. You should be able to see a 'Join' button as shown in image Image 1 - client.

Open the console on the browser and after that click the 'Join' button you should be able to see that the server is emitting data to the client as seen in Image 2 - client console

If you look at the terminal on server code you should be able to see the client is emitting the data to the server as seen in image Image 3 - server terminal

Rooms

From time to time it is necessary to separate certain users so that we can only send events to specific users. One good example of how rooms can be used is a chat room. A chat room can be made for one or more people and only users who are in a particular room can receive the specific events.

Updated server code with rooms.

// Listen for socket.io connections
io.on('connection', (socket) => {
  // Listen for socket.io connections
  io.on('connection', (socket) => {
    console.log('New connection created');

    // Get the auth token provided on handshake.
    const token = socket.handshake.auth.token;
    console.log('Auth token', token);

    try {
      // Verify the token here and get user info from JWT token.
    } catch (error) {
      socket.disconnect(true);
    }

    // A client is disconnected.
    socket.on('disconnect', () => {
      console.log('A user disconnected');
    });

    // Read message recieved from client.
    socket.on('message_from_client', (data) => {
      console.log('message_from_client: ', data);
    });

    // Send a message to the connected client 5 seconds after the connection is created.
    setTimeout(() => {
      socket.emit('message_from_server', `Message: ${Math.random()}`);
    }, 5_000);
  });

  /**
   * New code
   */
  // Get the room number from the client.
  const roomNumber: string = socket.handshake.query.roomNumber as string;
  // Join room for specific users.
  const room = `room-userId-${roomNumber}`;
  socket.join(room);

  // Emit to room by room number.
  setTimeout(() => {
    io.to(room).emit('room-userId', `You are in room number: ${roomNumber}`);
  }, 2_000);

  // Add more socket event handlers here
});

The idea is to get the room number from the client and join specific users to specific rooms. After a user joins a specific room he will recieve events whenever we emit data to specific rooms.

// Get the room number from the client.
const roomNumber: string = socket.handshake.query.roomNumber as string;
// Join room for specific users.
const room = `room-userId-${roomNumber}`;
socket.join(room);

// Emit to room by room number.
setTimeout(() => {
  socketIo.to(room).emit('room-userId', `You are in room number: ${roomNumber}`);
}, 2_000);

On the client side, let's add input where users will be able to enter a room number and send the room number to the server side after the user presses the join button.

<!-- Add text input field next to 'Join' button -->
<input type="text" placeholder="Room number" id="roomId" />
<button onclick="establishConnection()">Join</button>

// Update connection for Socket.
const socketIo = io('http://localhost:4000', {
  auth: {
    token: 'json-web-token',
  },
  query: {
    roomNumber: document.getElementById('roomId').value, // <- new code
  },
});

Now let's open two tabs of the client application and let's join the same room. People from the same room will always see when someone joins the room as shown in the image Image 4 - joining the rooms

Wrapping up

In this tutorial we learned what are websockets and what are the advantages of using websockets instead of HTTP for real time communication and we learned that the Socket.IO is most popular option for using websockets with Node.js. Socket.IO is widely used by most popular companies like Microsoft, Trello etc. We learned how to create an Express server using Socket.IO and how to use it in client side. We learned how to send JWT tokens on Socket.IO handshake and how to send any additional data while initial communication is created. We also saw what the benefits are and why rooms in Socket.IO are used for.

Data validation with Express and Node.js

admirnisic — Tue, 12 Apr 2022 20:21:42 +0000

Github repository

In this tutorial we will learn how and why we need validating the data which is arriving on the API. Data validation is an essential part of an application whether the task is to collect information, analyze the data, prepare the data for presentation and many other use cases. It is important to verify the incoming data from the start because if the unwanted data is dragged on further through the application then it can happen that we have data that is not accurate.
While data validation is a critical and important step in any data workflow, unfortunately it is often skipped over. Data validation requires more time and thus it slows down the work, however, it is esential because it will help us to create a cleaner data flow.

Nowadays, data validation is becoming easier to implement thanks to the many libraries that exist. There are many libraries out there but I will mention only few of them: class-validator, joi, fastst-validator.
Just to mention that NestJS, which is a popular framework for building scalable Node.js applications, uses class-validator. Moleculer is another framework for building server side applications and is using fast-validator as default validator.

What is important to note is that some validators work with the json-schema (joi, fastest-validator) of objects while some validators work using classes by adding decorators (class-validator).

I personally think it is better to use a class based validator with the TypeScript language because it is not necessary to write classes and json-objects separately but we can use existing classes by adding decorators. Such is the case with class-validator and this is the library we will use in this tutorial.

Modeling

We will create a simple model for creating notes. Let's create it inside src/models/note as note.db.ts. In this file we will keep models for DB and for validation as well.

export class Note {
  _id: string;

  title: string;
  body: string;

  authorId: string;

  createdAt: string;
  updatedAt: string;
}

Code for mongoose (inside same file).

import { model, Model, Schema } from 'mongoose';
import { Note } from './note.model';

const NoteSchema = new Schema<Note>(
  {
    _id: { type: String, required: true },

    title: { type: String, required: true },
    body: { type: String, required: true },

    authorId: { type: String, required: true },
  },
  { collection: 'note', timestamps: true }
);

export const NoteModel: Model<Note> = model('note', NoteSchema);

We need to install class-validator library and add experimentalDecorators in tsconfig.json file

npm install --save class-validator

{
  "compilerOptions": {
    "target": "es2016",
    "module": "commonjs",
    "esModuleInterop": true,
    "skipLibCheck": true,
    "outDir": "./dist",
    "rootDir": "./src",
    "baseUrl": "./",
    "experimentalDecorators": true // <- new option
  }
}

Now we can create models for validation and if we look at the code below, we will see that we have a couple of models.

Note is a basic model which is used for mongoose for creating its schema.
NoteCreate model is a model which is used to create data for MongoDB.
NoteCreateAPI is a validation model which is the data that we expect to come to the API.

We will add the models inside the same file note.db.ts we previously created.

import { IsString, MaxLength, MinLength } from 'class-validator';

// Actual model.
export class Note {
  _id: string;

  title: string;
  body: string;

  authorId: string;

  createdAt: string;
  updatedAt: string;
}

// Model for creating item in database.
export type NoteCreate = Pick<Note, '_id' | 'title' | 'body' | 'authorId'>;

// Validation model which comes to the API.
export class NoteCreateAPI implements Pick<Note, 'title' | 'body'> {
  @IsString()
  @MinLength(10)
  @MaxLength(500)
  title: string;

  @IsString()
  @MinLength(100)
  @MaxLength(5_000)
  body: string;
}

If we look at the NoteCreateAPI model we will see that we picked only title and body properties which are required in order to create the note. We will focus only on the property title. We added 3 decorators:

@IsString() - value must be of type string.
@MinLength(10) - value must be at least 10 characters long.
@MaxLength(500) - value must be at most 500 characters long.

I have added only some basic decorators but there is a great flexibility on how we want that model to look. More about what our model can look like and what parameters we can include we can see the documentation from the library: class-validator documentation.

We will now create a POST API method and send data to that route.

NOTE: The route is protected with authMiddleware we created in one of the previous tutorials. Align all the imports in server.ts file.

app.post('/note', authMiddleware, async (req: Request, res: Response, next: NextFunction) => {
  // data from the token that is verified
  const noteNew = new NoteCreateAPI();
  noteNew.title = req.body.title;
  noteNew.body = req.body.body;

  // verify input parameters
  const errors = await validate(noteNew);
  if (errors.length) {
    throw new ErrorException(ErrorCode.ValidationError, errors); // <- new ErrorCode enum
  }

  // create note data
  const tokenData: { _id: string; email: string } = req.body.tokenData;
  const noteCreate: NoteCreate = {
    _id: ulid(),
    title: noteNew.title,
    body: noteNew.body,

    authorId: tokenData._id,
  };

  const created = await NoteModel.create(noteCreate);
  res.send({ note: created });
});

NOTE: In error-code.ts add additional enum in switch case for status code.

    switch (code) {
      case ErrorCode.Unauthenticated:
        this.status = 401;
        break;
      case ErrorCode.MaximumAllowedGrade:
        this.status = 400;
        break;
      case ErrorCode.AsyncError:
        this.status = 400;
        break;
      case ErrorCode.ValidationError: // <- new
        this.status = 422;
        break;
      case ErrorCode.NotFound:
        this.status = 404;
        break;
      default:
        this.status = 500;
        break;
    }

Now that everything is prepared, we can check what happens when we send data which is not valid and when we send data which is valid.

If we look at Example 1 we will see that the field title field is missing and that field body is missing the character length.

In Example 2 we can see that the field title is present but the character lenght is not met, same is with field body.

Finally in Example 3 we can see that both title and body meet the requirements and that we have successfully created a note.

Wrapping up

In this tutorial we learned why it is important to have a validator in our application and we briefly looked at which libraries we can use in the Node.js application. We mentioned why I chose the class-validator library and finally created the code implementation that demonstrates a couple of examples of failed and one example of successful validation.

Authentification in NodeJS with Express using JWT

admirnisic — Tue, 08 Feb 2022 19:56:33 +0000

Github repository

I don't think it's necessary to explain why we need to have an authentication system in an application at all. You've probably heard of the terms authentication and authorization and I have to point out that these words have different meanings.
"Authentication is the act of validating that users are whom they claim to be. This is the first step in any security process. " Okta.com
"Authorization in system security is the process of giving the user permission to access a specific resource or function. This term is often used interchangeably with access control or client privilege." Okta.com

In this tutorial we will learn how to make an authentication system using JWT.

Database models

We will first have to deal with the database because we need to store user data somewhere. We need to store email and hashed password which will be used later for the sign in process. For this tutorial we will use NoSQL MongoDB database and we will also use mongoose. Mongoose is a MongoDB object modeling tool which is designed to work in an asynchronous environment and supports both promises and callbacks.

We will install the necessary packages:

npm install --save mongoose
npm install --save-dev @types/mongoose

After the packages are installed, we can start making the model. We will create a model for the user who will have the fields _id, email, name and password. We will also create a unique email index so that there are no two users with the same email in our database.
We will create the user.db.ts file in folder src/models/user.

import { model, Model, Schema } from 'mongoose';

export interface IUser {
  _id: string;
  email: string;
  password: string;
  name: string;
}

const IUserSchema = new Schema<IUser>(
  {
    _id: { type: String, required: true },
    email: {
      type: String,
      required: true,
      lowercase: true,
      index: true,
      unique: true,
    },
    name: { type: String, required: true },
    password: { type: String, required: true },
  },
  { collection: 'user', timestamps: true }
);

export const UserModel: Model<IUser> = model('user', IUserSchema);

Now lets create a connection to the MongoDB database via mongoose. We will call the file mongoose.index.ts and place it foldersrc/models.

Note: We need to have a MongoDB database running in order to connect to it. If you use docker you can find the docker-compose.yml file on github which link is provided in this tutorial and just run docker-compose up -d.

import mongoose, { Connection } from 'mongoose';

let mongooseConnection: Connection = null;
export async function connect(): Promise<void> {
  try {
    mongoose.connection.on('connecting', () => {
      console.log(`MongoDB: connecting.`);
    });
    mongoose.connection.on('connected', () => {
      console.log('MongoDB: connected.');
    });
    mongoose.connection.on('disconnecting', () => {
      console.log('MongoDB: disconnecting.');
    });
    mongoose.connection.on('disconnected', () => {
      console.log('MongoDB: disconnected.');
    });

    if (mongoose.connection.readyState !== 1 && mongoose.connection.readyState !== 2) {
      const conn = await mongoose.connect('mongodb://localhost:27017/ts-tutorial', { // <- replace connection string if necessary
        autoIndex: true,
        serverSelectionTimeoutMS: 5000,
      });
      mongooseConnection = conn.connection;
    }
  } catch (error) {
    console.log(`Error connecting to DB`, error);
  }
}

Now in the server.ts file we can call the method for connecting to the database:

connect();

Align the imports for connect function.

If the application is successfully connected to the database then we should get the messages from log:

MongoDB: connecting.
Application started on port 3000!
MongoDB: connected

Sign up process

We will first create an endpoint to which we will send data to create a new user. We will add the new route in the server.ts file. Email, name and password fields are required (we will not do the validation of parameters). After that, we must first check if there is an existing user with the same email and only after we determine that the user does not exist, can we proceed further.
The next step is to make a hash of the plain password because the plain password is never stored in the database. So when we create a new user we take his plain password, make a hash and keep the hash in the database. We will need the hashed password later for the sign in process.

Required npm packages:

npm install --save ulid
npm install --save bcrypt
npm install --save-dev @types/bcrypt

app.post('/sign-up', async (req: Request, res: Response, next: NextFunction) => {
  const { email, name, password } = req.body;
  // check if user exists
  const userExists = await UserModel.findOne({ email: email });
  if (!!userExists) {
    next(new ErrorException(ErrorCode.DuplicateEntityError, { email }));
  }

  // generate password hash
  const hash = passwordHash(password);
  const newUser: IUser = {
    _id: ulid(),
    email,
    name,
    password: hash,
  };
  const created = await UserModel.create(newUser);
  res.send({ done: true });
});

Note: Add the code in server.ts file before the routes for encoding the data that is being sent to our application from the client side.

const app = express();
app.use(
  express.urlencoded({
    extended: true,
  })
);

app.use(express.json());

We used the bcrypt library to create a hash from a plain password. Create a file crypto.lib.ts in a new folder src/lib. The code for hashing and comparing plain and hashed passwords:

import bcrypt from 'bcrypt';

export const passwordHash = (plainPassword: string): string => {
  const hash = bcrypt.hashSync(plainPassword, 10);
  return hash;
};

export const comparePassword = (plainPassword: string, passwordHash: string): boolean => {
  const compared = bcrypt.compareSync(plainPassword, passwordHash);
  return compared;
};

In the code above, you can see that we have two functions. The passwordHash function will hash a plain password.
The comparePassword function will check that the plain password entered is the same as the hash from the database. We will need this method later for the login form.

Additional steps:

align imports

import express, { NextFunction } from 'express';
import { Request, Response } from 'express';
import { errorHandler } from './error-handler/error-handler';
import { ErrorException } from './error-handler/error-exception';
import { ErrorCode } from './error-handler/error-code';
import { connect } from './models/mongoose.index';
import { IUser, UserModel } from './models/user/user.db'; // <- new import
import { passwordHash } from './lib/crypto.lib'; // <- new import
import { ulid } from 'ulid'; // <- new import

add new enum value ErrorCode.DuplicateEntityError

export class ErrorCode {
  public static readonly Unauthenticated = 'Unauthenticated';
  public static readonly NotFound = 'NotFound';
  public static readonly MaximumAllowedGrade = 'MaximumAllowedGrade';
  public static readonly AsyncError = 'AsyncError';
  public static readonly DuplicateEntityError = 'DuplicateEntityError'; // <- new enum value
  public static readonly UnknownError = 'UnknownError';
}

If we have successfully created a user in the database, the next step is to create a JWT when the user tries to sign in.

Sign in process

As we said in the introduction, we will use the jsonwebtoken package and for that we need to install the packages:

npm install --save jsonwebtoken
npm install --save-dev @types/jsonwebtoken

Actually how does it work? It is necessary to create a route for sign in where it will be necessary to enter email and password.

We will first check if there is a user with the provided email and if there is one, then we will take the password hash that is saved in the database. It is necessary to check whether the plain password from the login form agrees with the hash password from the database using the comparePassword method. If the method returns true then the user has entered a good password, otherwise the method will return false.

After that, it is necessary to generate jsonwebtoken through the mentioned library. We will generate the JWT with the help of a secret key which we keep in our application and the client should not be aware of the secret key. We will generate that jsonwebtoken string and return that token to the client application.

Create new file auth.lib.ts inside src/lib folder with following code:

import { IUser } from '../models/user/user.db';
import jwt from 'jsonwebtoken';
import { ErrorException } from '../error-handler/error-exception';
import { ErrorCode } from '../error-handler/error-code';

const jwtKey = 'keyyyy';

export const generateAuthToken = (user: IUser): string => {
  const token = jwt.sign({ _id: user._id, email: user.email }, jwtKey, {
    expiresIn: '2h',
  });

  return token;
};

export const verifyToken = (token: string): { _id: string; email: string } => {
  try {
    const tokenData = jwt.verify(token, jwtKey);
    return tokenData as { _id: string; email: string };
  } catch (error) {
    console.error('Error verifying token', error);
    throw new ErrorException(ErrorCode.Unauthenticated);
  }
};

app.post('/sign-in', async (req: Request, res: Response, next: NextFunction) => {
  const { email, password } = req.body;
  // check if user exists
  const userExists = await UserModel.findOne({ email: email });
  if (!userExists) {
    next(new ErrorException(ErrorCode.Unauthenticated));
  }

  // validate the password
  const validPassword = comparePassword(password, userExists.password);
  if (!validPassword) {
    next(new ErrorException(ErrorCode.Unauthenticated));
  }

  // generate the token
  const token = generateAuthToken(userExists);

  res.send({ token });
});

Additional steps:

align imports in server.ts file

...
import { ulid } from 'ulid';
import { generateAuthToken } from './lib/auth.lib'; // <- new import

Authentication middleware

We will create one middleware called authMiddleware which we will put on the routes where we need to have protection and whose job will be to check if the JWT that was generated is valid. authMiddleware function is just a middleware function which will get a token from the header and check its validation. We can check the validation of the token with the function verifyToken which is placed inside our middleware. We will create file auth.middleware.ts inside src/middlewares folder.

import { Request, Response, NextFunction } from 'express';
import { ErrorCode } from '../error-handler/error-code';
import { ErrorException } from '../error-handler/error-exception';
import { verifyToken } from '../lib/auth.lib';

export const authMiddleware = (req: Request, res: Response, next: NextFunction) => {
  const auth = req.headers.authorization;
  if (auth && auth.startsWith('Bearer')) {
    const token = auth.slice(7);

    try {
      const tokenData = verifyToken(token);
      req.body.tokenData = tokenData;
      next();
    } catch (error) {
      console.error('Error verifying token', error);
      throw new ErrorException(ErrorCode.Unauthenticated);
    }
  } else {
    throw new ErrorException(ErrorCode.Unauthenticated);
  }
};

The client side is required to send the JWT token string in the header for each API call that requires authentication. Header with authorization token looks like:

Authorization: Bearer eyJhbGciOiJIUzI1NiIXVCJ9TJV...r7E20RMHrHDcEfxjoYZgeFONFh7HgQ

Protected route with middleware:

app.get('/protected-route', authMiddleware, (req: Request, res: Response, next: NextFunction) => {
  // data from the token that is verified
  const tokenData = req.body.tokenData;
  console.log('tokenData', tokenData);
  res.send('this is a protected route');
});

Align the imports in server.ts

Wrapping up

In this tutorial we covered how to create basic models with mongoose and MongoDB and how to connect to MongoDB instances. We also learned how to create a new user and save the user in the database and what is important, how to create a hash password using the bcrypt library. After saving the user, we showed how to create a sign in process and generate a token using the jsonwebtoken library. Finally, we demonstrated how to create one middleware to be placed on a route to protect certain routes.

Error handling in Node.js with Express

admirnisic — Sat, 07 Aug 2021 13:36:45 +0000

Github repository

Why do we need error handling at all?

Imagine that a client application (web, mobile...) is using the server. Sometimes we need to handle exceptions which appear in the application and we need to send a clear message to the user what is wrong. It is very important to have a working error handler inside the application in order to achieve better user experience and for many other reasons as well. Beside from the user experience, it is a good practice to catch errors in one place (all the errors go through the handler) so the developer can track the bugs/exceptions more easily.

Creating exceptions

An exception is created using the throw keyword inside the application.

throw Error('Error');

As soon as the application executes this line the normal flow is halted and the control is switched to the nearest exception handler. While in some other environments we can throw strings, objects etc, in Node.js we throw error objects. An error object is an object derived from Error or an instance of Error itself.

Throwing an error from derived Error object looks like:

class SomethingIsWrongError extends Error {
  constructor() {
    super('Something is wrong!');
  }
}
throw new SomethingIsWrongError();

Before we start creating our error handler we need to decide what is the right way to go. Most of my applications have supported/support multiple languages which means that the message needs to be translated into the language which the user has selected. We cannot show errors in English language if the user has selected Japanese language which means the error message needs to be translated somewhere. Either we translate the message on the server side or on the client side.

Server side translation
In order to translate the message on the server side we have to know to whom we are sending the exception in order to get the selected language from the user. Challenge of this approach is that a developer needs always to have the selected language of the user whenever an error message needs to be sent to the client side.
Client side translation
Other solution is to send an unique error code and any additional data if needed so the translation of exceptions should be done on the client side based on the code and this is the solution which I prefer.

The client side needs to know:

Status code.
Unique error code. Every error has its own unique code.
Metadata if any. If any additional dynamic data needs to be sent in order to translate the message like what is the maximum allowed input number etc.

In order to keep track of all the errors more easily, we need to create a class in which we will store all possible errors that we know about. When we throw an exception then we will refer to one of the codes found in that class.

Create a folder called error-handler in the root directory and this will be the place where we will create files for error handler logic. Create a file called error-code.ts with following code:

export class ErrorCode {
  public static readonly Unauthenticated = 'Unauthenticated';
  public static readonly NotFound = 'NotFound';
  public static readonly MaximumAllowedGrade = 'MaximumAllowedGrade';
  public static readonly AsyncError = 'AsyncError';
  public static readonly UnknownError = 'UnknownError';
}

We also need to have a model that we will return to the client. Create a file called error-model.ts inside error-handler folder with following code:

export class ErrorModel {
  /**
   * Unique error code which identifies the error.
   */
  public code: string;
  /**
   * Status code of the error.
   */
  public status: number;
  /**
   * Any additional data that is required for translation.
   */
  public metaData?: any;
}

And now we need to create the actual error exception object. Create a file called error-exception.ts inside error-handler folder with following code:

import { ErrorCode } from './error-code';

export class ErrorException extends Error {
  public status: number = null;
  public metaData: any = null;
  constructor(code: string = ErrorCode.UnknownError, metaData: any = null) {
    super(code);
    Object.setPrototypeOf(this, new.target.prototype);
    this.name = code;
    this.status = 500;
    this.metaData = metaData;
    switch (code) {
      case ErrorCode.Unauthenticated:
        this.status = 401;
        break;
      case ErrorCode.MaximumAllowedGrade:
        this.status = 400;
        break;
      case ErrorCode.AsyncError:
        this.status = 400;
        break;
      case ErrorCode.NotFound:
        this.status = 404;
        break;
      default:
        this.status = 500;
        break;
    }
  }
}

When we want to throw an error from our application we use exactly the class we created and one code from the available list of codes. We would throw an error like:

throw new ErrorException(ErrorCode.MaximumAllowedGrade, { max: 100 }); // object is optional

Error handler

Error handler is a special middleware in Node.js which takes 4 parameters. Regular route middleware takes 3 parameters: req, res and next. Error handler also takes these 3 parameters and one additional parameter which is the actual error. Those four parameters are (retrospectively):

Create file called error-handler.ts inside error-handler folder. The following handler will intercept all errors that occur in the application whether it is an exception that we know or an exception that we do not know. In order to recognize that it is an exception thrown by ourselves, we can recognize it by type of instance if (err instanceof ErrorException)

import { Request, Response, NextFunction } from 'express';
import { ErrorCode } from './error-code';
import { ErrorException } from './error-exception';
import { ErrorModel } from './error-model';

export const errorHandler = (err: Error, req: Request, res: Response, next: NextFunction) => {
  console.log('Error handling middleware called.');
  console.log('Path:', req.path);
  console.error('Error occured:', err);
  if (err instanceof ErrorException) {
    console.log('Error is known.');
    res.status(err.status).send(err);
  } else {
    // For unhandled errors.
    res.status(500).send({ code: ErrorCode.UnknownError, status: 500 } as ErrorModel);
  }
};

Now it is necessary to register this handler and we will register it as follows. The handler needs to be 'lowered' as far as possible in the application after all routes and other middlewares and handlers. If we specify routes or middlewares after registration of errorHandler then the error handler will not catch exceptions which appear in those routes or middlewares.

app.use(errorHandler); // registration of handler

app.listen(3000, () => {
  console.log('Application started on port 3000!');
});

Now we are ready to throw some errors. Adjust the imports as needed.

app.get('/throw-unauthenticated', (req: Request, res: Response, next: NextFunction) => {
  throw new ErrorException(ErrorCode.Unauthenticated);
  // or
  // next(new ErrorException(ErrorCode.Unauthenticated))
});
app.get('/throw-maximum-allowed-grade', (req: Request, res: Response, next: NextFunction) => {
  throw new ErrorException(ErrorCode.MaximumAllowedGrade, { grade: Math.random() });
  // or
  // next(new ErrorException(ErrorCode.MaximumAllowedGrade, { grade: Math.random() }))
});
app.get('/throw-unknown-error', (req: Request, res: Response, next: NextFunction) => {
  const num: any = null;
  // Node.js will throw an error because there is no length property inside num variable
  console.log(num.length);
});

Add /* eslint-disable @typescript-eslint/no-unused-vars */ at the top of the file to ignore linter errors for not using variables like req: Request

If you look at the code above, you will see that we have 2 known exceptions and one unknown. When we want to throw an exception from a route we can do it with the throw keyword or by calling the next function with an actual exception. Error handler will catch both exceptions. However, when it comes to async logic then it will be solved in another way which we will cover next.

Exceptions with promises

By Exress documentation:
Handling sync code:
Errors that occur in synchronous code inside route handlers and middleware require no extra work. If synchronous code throws an error, then Express will catch and process it. For example:

app.get('/', function (req, res) {
  throw new Error('BROKEN'); // Express will catch this on its own.
});

Handling async code in Express version 4:
For errors returned from asynchronous functions invoked by route handlers and middleware, you must pass them to the next() function, where Express will catch and process them. For example:

app.get('/', function (req, res, next) {
  fs.readFile('/file-does-not-exist', function (err, data) {
    if (err) {
      next(err); // Pass errors to Express.
    } else {
      res.send(data);
    }
  });
});

Starting with Express 5, route handlers and middleware that return a Promise will call next(value) automatically when they reject or throw an error. For example:

app.get('/user/:id', async function (req, res, next) {
  // if error appears in getUserById, express will automatically throw an error
  const user = await getUserById(req.params.id);
  res.send(user);
});

Let's add code into our application for async code. The code will always throw an error and regarding if we are using express 4 or express 5, the application will catch the error.

const someOtherFunction = () => {
  const myPromise = new Promise((resolve, reject) => {
    setTimeout(() => {
      reject(new ErrorException(ErrorCode.AsyncError));
    }, 1000);
  });
  return myPromise;
};
app.get('/throw-async-await-error', async (req: Request, res: Response, next: NextFunction) => {
  // express 4
  // try {
  //   await someOtherFunction();
  // } catch (err) {
  //   next(err);
  //   // next line will not work as expected
  //   // throw err
  // }
  // express 5
  await someOtherFunction();
});

Wrapping up

In this tutorial we covered what exceptions are and how to throw an exception in application. We learned what we need to consider when handling exceptions in multi language applications. We learned to do everything necessary for the Node.JS application to successfully manage exceptions from creating necessary classes to creating a handler and registering it. And finally we learned how to throw exceptions and what to take care of when throwing exceptions in async or sync blocks.

Comming up: Authentication with JWT.

Create new Node.js application with Express, TypeScript, Nodemon and ESLint

admirnisic — Sun, 06 Jun 2021 17:53:35 +0000

Github repository

Node.js is an open-source and cross-platform runtime environment and a very popular tool for building applications. It runs on V8 JavaScript engine which makes it highly performant. It was released in 2009 and since then it is getting more and more popular. One of the advantages is that it uses JavaScript programming language which means that the same programming language can be used for server-side and client-side applications. There are few popular frameworks which run on Node.js platform and most popular is Express. In this tutorial I am going to use Express framework.

A big positive change emerged when the TypeScript language was released. TypeScript is a language which builds on JavaScript and some of the advantages using TypeScript are:

Type checking.
Find bugs more easily while writing code.
Extraordinary tooling with IntelliSense.
TypeScript upholds interfaces.
Fully supported SOLID principles.

It is not hard to migrate your code from JavaScript to TypeScript and I strongly recommend using TypeScript in Node.js projects.

Creating an express project.

We are going to start with a new project. Initialize npm with command:

npm init -y

The above command will generate package.json file which is required for npm to install packages. Then install dependencies for Express and TypeScript.

npm install --save express
npm install --save-dev typescript ts-node @types/node @types/express

These are all the dependencies that needs to be installed in order to run an Express application with TypeScript. After the installation completes package.json file should look like this.

{
  "name": "nodejs-typescript",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "express": "^5.1.0"
  },
  "devDependencies": {
    "@types/express": "^5.0.3",
    "@types/node": "^24.1.0",
    "ts-node": "^10.9.2",
    "typescript": "^5.8.3"
  }
}

There is one additional step that needs to be done in order to run TypeScript code. Typescript requires a configuration file. We need to create a file called tsconfig.json which will be placed inside root directory. Generate a tsconfig.json file with command:

npx tsc --init

A file called tsconfig.json will be created which holds the configuration for TypeScript. Newly created file contains a lot of configuration but we will consider only few of them.

{
  "compilerOptions": {
    "target": "es2016",
    "module": "commonjs",
    "esModuleInterop": true,
    "skipLibCheck": true,
    "outDir": "./dist",
    "rootDir": "./src",
    "baseUrl": "./"
  }
}

target: specifies which ECMAScript version should be used in the project. The available versions are ES3 (default), ES5, ES2015, ES2016, ES2017, ES2018, ES2019, ES2020, or ESNEXT.

module: specifies which module manager to use in the generated JavaScript code. Available options are none, commonjs, amd, system, umd, es2015, es2020, or ESNext. The most common module manager and the default one is commonjs.

outDir: specifies where to output JavaScript code after build.

rootDir: specifies where the TypeScript files are located.

baseUrl: specifies what is the relative path when including files in the application.

esModuleInterop: this option is true by default; it controls interoperability between CommonJS and ES modules. It does this by creating namespace objects for all imports.

Create application files

Create a folder called src where the application files will be placed and then inside that folder create a file called server.ts. File server.ts will be the starting point of the application.

import express from 'express';
import { Request, Response } from 'express';

const app = express();

app.get('/', (req: Request, res: Response) => {
  res.send('Application works!');
});

app.listen(3000, () => {
  console.log('Application started on port 3000!');
});

Now we are ready to start and build our project. We are going to add scripts for starting and building the application inside package.json file. Change your package.json script object to look like:

"scripts": {
    "start": "ts-node ./src/server.ts",
    "start:prod": "npm run build && node ./dist/server.js",
    "build": "npx tsc"
},

Building the application is pretty simple. We need to run command npm run build. TypeScript will consider tsconfig.json file for configuration. Earlier we set "outDir": "./dist", in configuration which means that TypeScript will build application inside dist folder.

We can run the application with commands:

npm run start or just npm start (development)
npm run start:prod (production)

After we start the application we can visit http://localhost:3000 in any browser and except that we see Application works!.

Nodemon

Nodemon is a tool which is widely used and which tracks changes and automatically restarts the application. If we do not use Nodemon then after each change we have to stop the application and run it again.

We can install Nodemon with command:

npm install --save-dev nodemon

Create nodemon.json configuration file inside root directory.

{
  "ignore": [".git", "node_modules", "dist"],
  "watch": ["./src"], // <- files inside folder to watch
  "exec": "npm start", // <- command that will be executed when nodemon starts
  "ext": "ts" // <- files to watch
}

Remove the comments of course. Add additional script for nodemon inside package.json scripts.

"start:nodemon": "nodemon --exec ts-node ./src/server.ts",

Now we can start watching the application with command npm run start:nodemon. Application will be restarted after we change any TypeScript file inside src folder.

Using ESLint with prettier

Eslint is a linter for JavaScript/TypeScript which finds problems inside your code. I cannot imagine writing TypeScript code without using ESLint. Previously I was using TSLint for TypeScript but now it is deprecated and ESLint should be used instead. Additional package which I would recommend to use is prettier together with ESLint.
NOTE: Prettier can be used also to automatically format your code (which I prefer) but we will not cover it in this tutorial.

Install dependencies:

npm install --save-dev eslint eslint-config-prettier eslint-plugin-prettier @typescript-eslint/parser @typescript-eslint/eslint-plugin prettier @eslint/js typescript-eslint

ESLint also uses configuration file. For this purpose create eslint.config.mjs file in root directory. We can arrange rules to our needs.

import globals from 'globals';
import pluginJs from '@eslint/js';
import tseslint from 'typescript-eslint';
import prettier from 'eslint-plugin-prettier';
import prettierConfig from 'eslint-config-prettier';

/** @type {import('eslint').Linter.Config[]} */
export default [
  { ignores: ['dist/**'] },
  { files: ['**/*.{ts}'] },
  {
    languageOptions: {
      globals: globals.browser,
    },
  },
  pluginJs.configs.recommended,
  ...tseslint.configs.recommended,
  prettierConfig, // Disables ESLint rules that conflict with Prettier
  {
    plugins: {
      prettier,
    },
    rules: {
      'prettier/prettier': 'error', // Treat Prettier errors as ESLint errors
      'max-len': 'off',
      'no-underscore-dangle': 'off',
      'arrow-body-style': 'off',
      'jsdoc/newline-after-description': 'off',
      '@typescript-eslint/no-explicit-any': 'off',
    },
  },
];

If we need to ignore any folder/file for ESLint then add .eslintignore file inside root directory with configuration. For demonstration purpose I added the file with content.

/dist

Now ESLint will ignore all files inside dist folder.

We added plugin for prettier as well so the linter will tell us when something is not formatted by the prettier configuration. Add .pretterrc file inside root directory and adjust it to your needs. My config looks like:

{
  "singleQuote": true,
  "trailingComma": "es5",
  "printWidth": 130
}

Now we are able to see if there is any error/warning inside our code. Restart your IDE if needed.

Wrapping up

In this tutorial we covered how to use Node.js project with TypeScript language from scratch. We also covered how to build and run the application. Nodemon is used to restart the application whenever a change in code happens. And finally we enabled linter with ESLint and configured it to our needs together with prettier.